Nov 02 11:53:40.650609 ip-10-0-141-170 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 11:53:40.650621 ip-10-0-141-170 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 11:53:40.650628 ip-10-0-141-170 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 11:53:40.650852 ip-10-0-141-170 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 11:53:50.679645 ip-10-0-141-170 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 11:53:50.679663 ip-10-0-141-170 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 27f4c9f6d5a9473b94f488ab9c22ab6b -- Nov 02 11:55:42.208782 ip-10-0-141-170 systemd[1]: Starting Kubernetes Kubelet... Nov 02 11:55:42.584486 ip-10-0-141-170 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 11:55:42.584486 ip-10-0-141-170 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 11:55:42.584486 ip-10-0-141-170 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 11:55:42.584486 ip-10-0-141-170 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 11:55:42.588549 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588470 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:55:42.588549 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588549 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:55:42.588549 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588552 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588556 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588559 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588562 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588565 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588567 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588570 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588573 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588575 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588578 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588580 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588583 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588586 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588589 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588592 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588594 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588597 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588599 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588602 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:55:42.588648 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588604 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588614 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588617 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588619 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588622 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588624 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588627 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588630 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588632 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588635 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588637 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588639 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588642 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588645 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588648 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588651 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588654 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588657 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588660 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588662 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:55:42.589080 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588665 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588668 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588670 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588673 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588676 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588679 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588681 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588684 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588687 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588689 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588692 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588695 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588698 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588700 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588709 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588712 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588714 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588717 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588720 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588724 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:55:42.589548 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588727 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588729 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588731 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588734 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588737 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588739 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588742 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588745 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588747 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588750 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588752 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588755 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588758 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588760 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588763 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588765 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588768 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588770 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588773 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588776 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:55:42.590010 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588778 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588781 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588783 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588786 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588788 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588791 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588793 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588801 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588804 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588806 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588810 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588813 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588815 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588820 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588824 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588828 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588832 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588835 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:55:42.590475 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588838 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588841 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588843 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588846 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588848 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588851 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.588854 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589721 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589728 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589731 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589734 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589736 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589739 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589742 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589745 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589747 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589750 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589753 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589755 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:55:42.590933 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589758 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589760 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589763 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589771 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589774 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589777 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589780 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589783 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589786 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589788 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589791 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589793 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589796 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589799 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589802 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589804 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589807 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589810 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589812 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:55:42.591401 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589815 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589817 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589821 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589824 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589827 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589830 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589833 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589836 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589838 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589841 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589844 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589846 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589849 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589852 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589854 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589857 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589861 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589872 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589876 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:55:42.591860 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589878 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589881 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589884 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589887 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589889 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589891 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589894 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589896 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589900 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589902 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589905 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589907 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589910 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589912 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589915 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589917 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589920 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589928 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589930 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589933 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:55:42.592317 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589935 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589938 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589940 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589943 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589945 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589948 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589951 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589954 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589956 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589959 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589961 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589969 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589972 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589975 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589978 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589980 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589983 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589986 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589989 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589991 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:55:42.592808 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589994 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589997 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.589999 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590002 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590005 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590008 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590010 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590013 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590016 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590019 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590022 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590025 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590031 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590033 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590036 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590039 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590602 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590607 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590610 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590613 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:55:42.593265 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590616 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590618 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590621 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590629 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590632 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590635 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590637 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590640 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590642 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590645 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590648 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590650 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590654 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590657 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590660 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590662 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590665 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590667 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590670 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590673 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590675 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:55:42.593748 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590678 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590682 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590685 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590687 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590690 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590693 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590696 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590698 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590701 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590703 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590706 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590709 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590711 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590714 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590716 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590724 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590727 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590730 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590732 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:55:42.594227 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590735 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590737 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590740 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590743 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590747 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590749 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590752 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590754 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590757 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590760 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590762 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590765 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590767 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590770 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590772 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590775 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590778 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590781 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590783 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590786 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:55:42.594690 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590789 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590791 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590794 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590796 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590799 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590801 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590804 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590806 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590809 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590817 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590820 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590822 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590825 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590827 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590829 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590832 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590836 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590838 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590841 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590844 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:55:42.595182 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590846 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590849 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590851 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590854 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590856 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590859 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590861 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590863 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590866 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590868 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590871 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590873 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590881 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590884 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590887 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590889 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590891 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590894 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590897 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:55:42.595664 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590902 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590906 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.590908 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592594 2582 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592609 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592618 2582 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592623 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592627 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592631 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592635 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592639 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592643 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592646 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592649 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592652 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592655 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592658 2582 flags.go:64] FLAG: --cgroup-root="" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592661 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592664 2582 flags.go:64] FLAG: --client-ca-file="" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592667 2582 flags.go:64] FLAG: --cloud-provider="external" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592670 2582 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592677 2582 flags.go:64] FLAG: --cluster-domain="" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592681 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592684 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 11:55:42.596116 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592687 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592690 2582 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592695 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592697 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592700 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592704 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592706 2582 flags.go:64] FLAG: --contention-profiling="false" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592710 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592712 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592718 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592721 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592725 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592734 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592737 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592740 2582 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592743 2582 flags.go:64] FLAG: --enable-server="true" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592746 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592757 2582 flags.go:64] FLAG: --event-burst="100" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592760 2582 flags.go:64] FLAG: --event-qps="50" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592763 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592766 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592769 2582 flags.go:64] FLAG: --eviction-hard="" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592773 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592777 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592780 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 11:55:42.596721 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592783 2582 flags.go:64] FLAG: --eviction-soft="" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592786 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592789 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592792 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592795 2582 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592798 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592801 2582 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592804 2582 flags.go:64] FLAG: --feature-gates="" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592808 2582 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592811 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592814 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592817 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592821 2582 flags.go:64] FLAG: --healthz-port="10248" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592824 2582 flags.go:64] FLAG: --help="false" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592827 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592831 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592835 2582 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592838 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592842 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592845 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592854 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592857 2582 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592860 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 11:55:42.597356 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592863 2582 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592866 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592869 2582 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592872 2582 flags.go:64] FLAG: --kube-reserved="" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592875 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592878 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592881 2582 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592884 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592887 2582 flags.go:64] FLAG: --lock-file="" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592890 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592893 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592896 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592902 2582 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592905 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592908 2582 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592911 2582 flags.go:64] FLAG: --logging-format="text" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592914 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592917 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592920 2582 flags.go:64] FLAG: --manifest-url="" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592923 2582 flags.go:64] FLAG: --manifest-url-header="" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592933 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592936 2582 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592940 2582 flags.go:64] FLAG: --max-pods="110" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592943 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592946 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 11:55:42.597908 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592951 2582 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592954 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592957 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592960 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592963 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592975 2582 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592978 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592981 2582 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592985 2582 flags.go:64] FLAG: --pod-cidr="" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592987 2582 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592990 2582 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592993 2582 flags.go:64] FLAG: --pods-per-core="0" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592996 2582 flags.go:64] FLAG: --port="10250" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.592999 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593002 2582 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b26846743edd24ea" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593005 2582 flags.go:64] FLAG: --qos-reserved="" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593008 2582 flags.go:64] FLAG: --read-only-port="10255" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593011 2582 flags.go:64] FLAG: --register-node="true" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593014 2582 flags.go:64] FLAG: --register-with-taints="" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593018 2582 flags.go:64] FLAG: --registry-burst="10" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593021 2582 flags.go:64] FLAG: --registry-qps="5" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593023 2582 flags.go:64] FLAG: --reserved-cpus="" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593026 2582 flags.go:64] FLAG: --reserved-memory="" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593030 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593033 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593036 2582 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 11:55:42.598512 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593039 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593042 2582 flags.go:64] FLAG: --runonce="false" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593045 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593048 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593051 2582 flags.go:64] FLAG: --seccomp-default="false" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593056 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593060 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593064 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593067 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593070 2582 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593073 2582 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593076 2582 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593084 2582 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593087 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593090 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593093 2582 flags.go:64] FLAG: --system-cgroups="" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593096 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593101 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593104 2582 flags.go:64] FLAG: --tls-cert-file="" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593107 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593113 2582 flags.go:64] FLAG: --tls-min-version="" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593116 2582 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593119 2582 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593122 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593125 2582 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 11:55:42.599418 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593128 2582 flags.go:64] FLAG: --v="2" Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593132 2582 flags.go:64] FLAG: --version="false" Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593136 2582 flags.go:64] FLAG: --vmodule="" Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593140 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593143 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593345 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593349 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593353 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593357 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593359 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593362 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593364 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593368 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593373 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593385 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593389 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593393 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593395 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593398 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593401 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:55:42.600143 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593410 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593413 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593416 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593418 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593421 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593424 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593426 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593429 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593432 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593434 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593437 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593440 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593442 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593445 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593447 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593450 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593452 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593455 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593458 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593460 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:55:42.600649 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593463 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593465 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593468 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593470 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593475 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593478 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593481 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593483 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593486 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593489 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593491 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593505 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593508 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593517 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593520 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593522 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593525 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593527 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593530 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:55:42.601125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593532 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593535 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593538 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593541 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593543 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593546 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593548 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593551 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593554 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593556 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593559 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593562 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593564 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593567 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593569 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593572 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593574 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593578 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593582 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593585 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:55:42.601658 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593587 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593590 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593593 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593595 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593597 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593600 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593603 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593613 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593617 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593621 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593624 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593627 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593630 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593633 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593635 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593638 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593640 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593643 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593646 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:55:42.602125 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593649 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593651 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593654 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593658 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593662 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593665 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593667 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593670 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593672 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593675 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593679 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593682 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593685 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.593713 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593762 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593767 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:55:42.602585 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593770 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593773 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593775 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593778 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593780 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593784 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593787 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593790 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593793 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593795 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593798 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593800 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593803 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593805 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593808 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593810 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593813 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593816 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593818 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593821 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:55:42.602961 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593823 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593826 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593829 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593831 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593834 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593836 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593840 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593842 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593845 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593849 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593851 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593854 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593856 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593859 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593861 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593864 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593866 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593869 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593872 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593874 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:55:42.603430 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593877 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593880 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593882 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593885 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593887 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593890 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593892 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593895 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593897 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593900 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593903 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593905 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593908 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593910 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593913 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593916 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593918 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593920 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593923 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593925 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:55:42.603916 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593928 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593931 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593933 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593936 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593938 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593941 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593943 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593946 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593948 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593951 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593954 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593957 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593961 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593964 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593967 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593970 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593973 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593976 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593978 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:55:42.604437 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593981 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593983 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593986 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593989 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593991 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593994 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.593997 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594001 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594004 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594007 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594010 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594013 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594016 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594019 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594021 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594024 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594026 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594029 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594031 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594034 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:55:42.604910 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594037 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594039 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594041 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594044 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.594046 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.601415 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.601432 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.602082 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.602089 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602155 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602161 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602166 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602170 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602173 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602177 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602180 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602182 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602186 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602189 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:55:42.605371 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602192 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602195 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602198 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602200 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602203 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602206 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602208 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602211 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602213 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602216 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602219 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602221 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602224 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602226 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602229 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602231 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602234 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602238 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602241 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:55:42.605828 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602244 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602247 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602249 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602252 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602255 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602258 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602260 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602263 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602266 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602268 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602271 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602273 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602278 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602281 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602284 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602287 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602289 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602293 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602295 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:55:42.606280 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602298 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602301 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602303 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602306 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602309 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602312 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602314 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602317 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602320 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602322 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602325 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602328 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602331 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602334 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602337 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602340 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602342 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602345 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602348 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602351 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:55:42.606742 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602354 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602356 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602359 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602361 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602364 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602366 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602369 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602372 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602374 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602378 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602382 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602385 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602388 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602391 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602394 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602396 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602399 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602401 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602404 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:55:42.607202 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602407 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602409 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602412 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602416 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602418 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602421 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602424 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602427 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602430 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602433 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602435 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602438 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602440 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602443 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602446 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602448 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602451 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602454 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:55:42.607666 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602457 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.602485 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602664 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602669 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602673 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602676 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602679 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602683 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602686 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602689 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602692 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602695 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602698 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602701 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602704 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602706 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:55:42.608095 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602709 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602711 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602714 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602717 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602720 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602723 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602725 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602728 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602730 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602733 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602736 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602738 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602741 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602743 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602746 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602748 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602751 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602753 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602756 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602758 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:55:42.608478 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602761 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602763 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602766 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602769 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602775 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602778 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602780 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602783 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602786 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602788 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602791 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602793 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602796 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602798 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602801 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602804 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602806 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602809 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602811 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602814 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:55:42.609077 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602816 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602819 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602821 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602824 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602826 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602829 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602833 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602837 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602840 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602842 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602845 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602847 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602850 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602852 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602855 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602857 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602860 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602863 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602865 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:55:42.609562 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602868 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602871 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602873 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602876 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602878 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602881 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602883 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602886 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602888 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602891 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602895 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602898 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602901 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602904 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602907 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602909 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602912 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602915 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602917 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602920 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:55:42.610004 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602922 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602925 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602927 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602930 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602932 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602935 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602937 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602940 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602943 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602945 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602948 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602951 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.602954 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.602977 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603009 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603012 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:55:42.610480 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603015 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603017 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603020 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603022 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603025 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603029 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603031 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603034 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603036 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603039 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603042 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603044 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603047 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603049 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603052 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603055 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603057 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603060 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603062 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:55:42.610873 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603065 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603067 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603070 2582 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603073 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603075 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603078 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603080 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603083 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603085 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603100 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603103 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603105 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603108 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603110 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603113 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603115 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603118 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603121 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603123 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603126 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:55:42.611335 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603129 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603131 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603134 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603136 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603139 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603141 2582 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603144 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603146 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603150 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603154 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603157 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603160 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603163 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603166 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603169 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603171 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603174 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603176 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603179 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:55:42.611895 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603181 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603184 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603186 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603190 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603192 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603212 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603216 2582 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603219 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603222 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603225 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603230 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603232 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603236 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603240 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603243 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603247 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603250 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603252 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603255 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:55:42.612351 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603258 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603261 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603263 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603266 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603268 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603271 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603273 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603276 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603279 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603281 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603284 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603287 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603289 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603292 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603294 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603298 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603301 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603307 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603310 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603312 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:55:42.612841 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603315 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603318 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603320 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603324 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603327 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603331 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:42.603334 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.603425 2582 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.605363 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.606320 2582 server.go:1014] "Starting client certificate rotation" Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.606414 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.606444 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.608382 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 11:55:42.613404 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.610822 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 11:55:42.618104 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.618087 2582 log.go:25] "Validated CRI v1 runtime API" Nov 02 11:55:42.620956 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.620934 2582 log.go:25] "Validated CRI v1 image API" Nov 02 11:55:42.621812 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.621785 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 11:55:42.627330 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.627312 2582 fs.go:135] Filesystem UUIDs: map[15f751de-ad9e-45dc-b2b6-dd4cf415daba:/dev/nvme0n1p3 3923697a-5ca7-4ed4-be00-97f60c35a80f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 02 11:55:42.627376 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.627330 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 11:55:42.634241 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.634104 2582 manager.go:217] Machine: {Timestamp:2026-11-02 11:55:42.631308232 +0000 UTC m=+0.309788672 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109544 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23d1f5cd06312ed0d81395841974a9 SystemUUID:ec23d1f5-cd06-312e-d0d8-1395841974a9 BootID:27f4c9f6-d5a9-473b-94f4-88ab9c22ab6b Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:01:41:11:50:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:01:41:11:50:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e6:44:7b:77:86:a5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 11:55:42.635142 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.635113 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 11:55:42.635256 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.635224 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 11:55:42.638563 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.638535 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 11:55:42.638679 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.638565 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-170.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 11:55:42.638734 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.638681 2582 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 11:55:42.638734 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.638689 2582 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 11:55:42.638734 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.638712 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 11:55:42.639404 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.639394 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 11:55:42.640104 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.640094 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 11:55:42.643673 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.643661 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 11:55:42.643797 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.643788 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 11:55:42.644560 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.644550 2582 kubelet.go:493] "Attempting to sync node with API server" Nov 02 11:55:42.644608 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.644561 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 11:55:42.644608 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.644575 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 11:55:42.644608 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.644584 2582 kubelet.go:405] "Adding apiserver pod source" Nov 02 11:55:42.644608 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.644592 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 11:55:42.644879 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.644868 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 11:55:42.647081 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.647065 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 11:55:42.651056 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.651044 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 11:55:42.651812 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.651800 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 11:55:42.651855 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.651831 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 11:55:42.652022 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652011 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 11:55:42.652056 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652028 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 11:55:42.652056 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652034 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 11:55:42.652056 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652040 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 11:55:42.652056 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652050 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 11:55:42.652154 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652060 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 11:55:42.652554 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652545 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 11:55:42.652587 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652555 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 11:55:42.652587 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652562 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 11:55:42.652587 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652569 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 11:55:42.652587 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652578 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 11:55:42.652587 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.652586 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 11:55:42.653930 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.653920 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 11:55:42.653961 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.653934 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 11:55:42.657488 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.657475 2582 server.go:1263] "Started kubelet" Nov 02 11:55:42.657580 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.657554 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 11:55:42.657690 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.657641 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 11:55:42.657732 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.657714 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 11:55:42.658765 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.658747 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 11:55:42.658930 ip-10-0-141-170 systemd[1]: Started Kubernetes Kubelet. Nov 02 11:55:42.661244 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.661227 2582 server.go:317] "Adding debug handlers to kubelet server" Nov 02 11:55:42.663438 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.663417 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 11:55:42.663905 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.663884 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 11:55:42.664240 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.664214 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:42.664632 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.664510 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 11:55:42.664632 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.664530 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 11:55:42.664632 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.664596 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 11:55:42.664632 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.664635 2582 reconstruct.go:97] "Volume reconstruction finished" Nov 02 11:55:42.664846 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.664642 2582 reconciler.go:29] "Reconciler: start to sync state" Nov 02 11:55:42.665945 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.665901 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 11:55:42.666112 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.666098 2582 factory.go:55] Registering systemd factory Nov 02 11:55:42.666174 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.666115 2582 factory.go:223] Registration of the systemd container factory successfully Nov 02 11:55:42.666291 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.666280 2582 factory.go:153] Registering CRI-O factory Nov 02 11:55:42.666291 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.666291 2582 factory.go:223] Registration of the crio container factory successfully Nov 02 11:55:42.666361 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.666332 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 11:55:42.666414 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.666368 2582 factory.go:103] Registering Raw factory Nov 02 11:55:42.666414 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.666378 2582 manager.go:1199] Started watching for new ooms in manager Nov 02 11:55:42.667235 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.667219 2582 manager.go:319] Starting recovery of all containers Nov 02 11:55:42.678817 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.678801 2582 manager.go:324] Recovery completed Nov 02 11:55:42.682932 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.682917 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:55:42.684402 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.684377 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:55:42.684484 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.684419 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:55:42.684484 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.684435 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:55:42.684938 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.684921 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:55:42.685019 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.684925 2582 cpu_manager.go:226] "Starting" policy="none" Nov 02 11:55:42.685019 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.684973 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 11:55:42.685019 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.684993 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 11:55:42.685649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.685625 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:55:42.685743 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.685656 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:55:42.685743 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.685667 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:55:42.687104 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.687087 2582 policy_none.go:50] "Start" Nov 02 11:55:42.687104 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.687104 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 11:55:42.687237 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.687114 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 11:55:42.688944 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.688932 2582 policy_none.go:44] "Start" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.727743 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.728808 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.728826 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.728842 2582 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.728875 2582 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.733049 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.738297 2582 manager.go:341] "Starting Device Plugin manager" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.738353 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.738363 2582 server.go:87] "Starting device plugin registration server" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.738582 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.738593 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.738677 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.738745 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.738753 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.739115 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 11:55:42.740628 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.739162 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:42.808878 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.808860 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-170.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 11:55:42.808972 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.808849 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-170.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 11:55:42.809870 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.808849 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-170.us-west-1.compute.internal.18e438616af0cafe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-170.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-170.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 11:55:42.657448702 +0000 UTC m=+0.335929142,LastTimestamp:2026-11-02 11:55:42.657448702 +0000 UTC m=+0.335929142,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-170.us-west-1.compute.internal,}" Nov 02 11:55:42.829546 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.829521 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal"] Nov 02 11:55:42.829621 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.829572 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:55:42.830346 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.830331 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:55:42.830413 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.830357 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:55:42.830413 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.830370 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:55:42.832611 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.832599 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:55:42.832763 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.832741 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:42.832819 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.832783 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:55:42.833802 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.833779 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:55:42.833802 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.833799 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:55:42.833935 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.833810 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:55:42.833935 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.833823 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:55:42.833935 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.833846 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:55:42.833935 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.833825 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:55:42.835937 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.835903 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:42.835937 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.835925 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:55:42.836626 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.836612 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:55:42.836706 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.836641 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:55:42.836706 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.836652 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:55:42.839224 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.839210 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:55:42.839874 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.839859 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:55:42.839960 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.839882 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:55:42.839960 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.839891 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:55:42.839960 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.839907 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:42.868772 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.868752 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" node="ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:42.871226 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.871210 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" node="ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:42.880271 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.880256 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 11:55:42.880342 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.880289 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kvjnx" Nov 02 11:55:42.889710 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.889692 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 02 11:55:42.889710 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.889700 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 02 11:55:42.889858 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.889713 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-170.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 02 11:55:42.889858 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.889757 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-170.us-west-1.compute.internal.18e438616c8c154d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-170.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-170.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-170.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 11:55:42.684403021 +0000 UTC m=+0.362883467,LastTimestamp:2026-11-02 11:55:42.684403021 +0000 UTC m=+0.362883467,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-170.us-west-1.compute.internal,}" Nov 02 11:55:42.889967 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.889950 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 11:55:42.965261 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.965236 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ebeef05fd06ac44ab33873433450b74-config\") pod \"kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"1ebeef05fd06ac44ab33873433450b74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:42.965340 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.965264 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:42.965340 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.965280 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:42.975305 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:42.975287 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:42.975359 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:42.975307 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-170.us-west-1.compute.internal\": node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:43.065715 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.065591 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:43.065792 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.065651 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:43.065792 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.065747 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ebeef05fd06ac44ab33873433450b74-config\") pod \"kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"1ebeef05fd06ac44ab33873433450b74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:43.065792 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.065771 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:43.065792 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.065789 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:43.066625 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.065804 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ebeef05fd06ac44ab33873433450b74-config\") pod \"kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"1ebeef05fd06ac44ab33873433450b74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:43.092364 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:43.092321 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:43.171699 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.171678 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:43.174655 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.174637 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:43.192716 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:43.192697 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:43.292772 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:43.292746 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:43.393739 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:43.393676 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:43.494784 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:43.494766 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:43.595575 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:43.595557 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:43.606706 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.606691 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 11:55:43.606888 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:43.606809 2582 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aadda5a39e08548bea3cd7c71322c295-74c64663e2d0aa4a.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-170.us-west-1.compute.internal.18e438616c8c154d\": read tcp 10.0.141.170:40134->77.112.245.159:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-170.us-west-1.compute.internal.18e438616c8c154d default 5315 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-170.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-170.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-170.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 11:55:42 +0000 UTC,LastTimestamp:2026-11-02 11:55:42.833812277 +0000 UTC m=+0.512292725,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-170.us-west-1.compute.internal,}" Nov 02 11:55:43.696262 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:43.696211 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:43.797254 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:43.797221 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:43.858322 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.858304 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 11:55:43.861568 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.861543 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:55:43.862569 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.862541 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:55:43.862652 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.862577 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:55:43.862652 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.862589 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:55:43.885251 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.885218 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 11:50:43 +0000 UTC" deadline="2028-08-15 14:31:03.698233562 +0000 UTC" Nov 02 11:55:43.885251 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.885248 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15650h35m19.812988575s" Nov 02 11:55:43.897376 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:43.897356 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:55:43.975606 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:43.975553 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 11:55:44.065857 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.065827 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:44.085403 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.085383 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 11:55:44.134098 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.134080 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 11:55:44.135730 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.135711 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:55:44.189916 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.189897 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 11:55:44.199978 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.199959 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 11:55:44.314662 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:44.314633 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b5e76aab2a7b6af2007243418d856f.slice/crio-3175751de3169d589d580a270de77ebd96dc2d79e201f8781d78cab718800358 WatchSource:0}: Error finding container 3175751de3169d589d580a270de77ebd96dc2d79e201f8781d78cab718800358: Status 404 returned error can't find the container with id 3175751de3169d589d580a270de77ebd96dc2d79e201f8781d78cab718800358 Nov 02 11:55:44.315214 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:44.315193 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebeef05fd06ac44ab33873433450b74.slice/crio-1e2c0b22d038233964e850e642f73c6fd514e9f4ffcb67ad22f0eae85e4c4d7d WatchSource:0}: Error finding container 1e2c0b22d038233964e850e642f73c6fd514e9f4ffcb67ad22f0eae85e4c4d7d: Status 404 returned error can't find the container with id 1e2c0b22d038233964e850e642f73c6fd514e9f4ffcb67ad22f0eae85e4c4d7d Nov 02 11:55:44.320490 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.320473 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 11:55:44.647393 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.647331 2582 apiserver.go:52] "Watching apiserver" Nov 02 11:55:44.664864 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.664828 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 11:55:44.734483 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.734417 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" event={"ID":"14b5e76aab2a7b6af2007243418d856f","Type":"ContainerStarted","Data":"3175751de3169d589d580a270de77ebd96dc2d79e201f8781d78cab718800358"} Nov 02 11:55:44.735653 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.735630 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" event={"ID":"1ebeef05fd06ac44ab33873433450b74","Type":"ContainerStarted","Data":"1e2c0b22d038233964e850e642f73c6fd514e9f4ffcb67ad22f0eae85e4c4d7d"} Nov 02 11:55:44.768093 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.768072 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 11:55:44.768417 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.768398 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-4ncdp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-mmzdg","kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal","openshift-dns/node-resolver-nnpfg","openshift-image-registry/node-ca-np85r","openshift-multus/multus-w4h5q","openshift-network-diagnostics/network-check-target-s2fqn","kube-system/global-pull-secret-syncer-d5dpl","openshift-cluster-node-tuning-operator/tuned-gnt4x","openshift-ovn-kubernetes/ovnkube-node-2wrsc","kube-system/konnectivity-agent-5pnd2","openshift-multus/network-metrics-daemon-p2frl"] Nov 02 11:55:44.772556 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.772537 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-np85r" Nov 02 11:55:44.774993 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.774973 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.777340 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.777316 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:44.777452 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:44.777382 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:55:44.779763 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.779743 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.782104 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.782081 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.785216 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.785190 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:55:44.787592 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.787555 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:44.787681 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:44.787627 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:55:44.789977 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.789957 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:44.792370 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.792349 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.794869 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.794851 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:44.794952 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:44.794908 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:55:44.798130 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.798110 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:55:44.800751 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.800733 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.830896 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.830875 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4pvvq\"" Nov 02 11:55:44.830896 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.830887 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 11:55:44.831058 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.830887 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 11:55:44.831058 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.830922 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 11:55:44.832408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.832388 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 11:55:44.832646 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.832633 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 11:55:44.832735 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.832719 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gpdd6\"" Nov 02 11:55:44.837088 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.837069 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 11:55:44.837506 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.837489 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 11:55:44.837599 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.837547 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 11:55:44.837599 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.837560 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hx94g\"" Nov 02 11:55:44.867638 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.867620 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 11:55:44.876886 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.876861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-var-lib-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.876966 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.876895 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cni-binary-copy\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.876966 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.876924 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-daemon-config\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.876966 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.876949 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-konnectivity-ca\") pod \"konnectivity-agent-5pnd2\" (UID: \"fcda3ab4-1f37-49db-8eb9-2b83fe89d484\") " pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:44.877117 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.876973 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-systemd\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.877117 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.876997 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-etc-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.877117 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877022 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:44.877117 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877059 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cnibin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.877117 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877087 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-kubernetes\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.877117 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-lib-modules\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.877345 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877151 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.877345 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877178 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-systemd-units\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.877345 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877194 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-multus\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.877345 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877211 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-kubelet\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.877345 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-etc-kubernetes\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.877345 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877243 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kc7z\" (UniqueName: \"kubernetes.io/projected/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kube-api-access-8kc7z\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.877345 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtqw\" (UniqueName: \"kubernetes.io/projected/7e55d04a-613c-4e3a-bcc5-32fea4e50693-kube-api-access-djtqw\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:55:44.877345 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877308 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-netns\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877346 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877365 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-bin\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877388 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e55d04a-613c-4e3a-bcc5-32fea4e50693-host\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f547b23-0439-4cc3-9127-387407a34b31-hosts-file\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877518 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f547b23-0439-4cc3-9127-387407a34b31-tmp-dir\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877551 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-conf-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877578 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-registration-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877602 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-var-lib-kubelet\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5wc6\" (UniqueName: \"kubernetes.io/projected/291cad74-3a77-43f6-abb4-d28aaa5eb20c-kube-api-access-c5wc6\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877667 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-kubelet\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877685 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-tmp\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.877712 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877704 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhlnz\" (UniqueName: \"kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz\") pod \"network-check-target-s2fqn\" (UID: \"44d668e0-e534-4e0b-8d32-31ead2f678a2\") " pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877738 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-agent-certs\") pod \"konnectivity-agent-5pnd2\" (UID: \"fcda3ab4-1f37-49db-8eb9-2b83fe89d484\") " pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877752 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75mwz\" (UniqueName: \"kubernetes.io/projected/7f547b23-0439-4cc3-9127-387407a34b31-kube-api-access-75mwz\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877768 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-slash\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877790 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-node-log\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877826 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-script-lib\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877861 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877893 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-dbus\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877919 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-ovn\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877935 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-netd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877956 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877974 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-env-overrides\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.877994 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-socket-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878010 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6533ee11-4dc9-4c0f-b035-1200f74d869b-iptables-alerter-script\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878028 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e55d04a-613c-4e3a-bcc5-32fea4e50693-serviceca\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:55:44.878143 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878043 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-bin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878059 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-sys-fs\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878076 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-conf\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878105 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-run\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878129 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-tuned\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878146 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqh2j\" (UniqueName: \"kubernetes.io/projected/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-kube-api-access-xqh2j\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878161 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-sys\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878190 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-kubelet-config\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878230 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-netns\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-multus-certs\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878286 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878316 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-config\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878340 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovn-node-metrics-cert\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878365 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-socket-dir-parent\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878389 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6533ee11-4dc9-4c0f-b035-1200f74d869b-host-slash\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:55:44.878814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-modprobe-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878454 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-host\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878520 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-system-cni-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878544 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cnibin\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878575 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8686b\" (UniqueName: \"kubernetes.io/projected/05fb7da7-c7b3-412e-a14e-45d7c10d4216-kube-api-access-8686b\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878600 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878625 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbbqq\" (UniqueName: \"kubernetes.io/projected/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-kube-api-access-jbbqq\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878649 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-k8s-cni-cncf-io\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhq46\" (UniqueName: \"kubernetes.io/projected/6533ee11-4dc9-4c0f-b035-1200f74d869b-kube-api-access-hhq46\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878710 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysconfig\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878740 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-systemd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-log-socket\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878783 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-system-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-hostroot\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.879296 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878837 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bctf\" (UniqueName: \"kubernetes.io/projected/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-kube-api-access-6bctf\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.879772 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.878858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-device-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.880974 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.880959 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 11:55:44.881053 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.880985 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bjwgn\"" Nov 02 11:55:44.881375 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.881359 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 11:55:44.881452 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.881381 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 11:55:44.881452 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.881382 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x7vnq\"" Nov 02 11:55:44.881568 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.881543 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 11:55:44.881644 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.881632 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 11:55:44.881701 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.881678 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 11:55:44.881701 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.881686 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 11:55:44.881779 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.881689 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g4drg\"" Nov 02 11:55:44.881779 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.881723 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 11:55:44.881779 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.881742 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 11:55:44.885702 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.885688 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 11:55:44.885702 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.885691 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2rrmx\"" Nov 02 11:55:44.885799 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.885701 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mvmxd\"" Nov 02 11:55:44.886342 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.886330 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 11:55:44.886398 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.886346 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 11:55:44.886398 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.886352 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 11:55:44.886398 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.886381 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 11:55:44.888201 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.888180 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 11:55:44.888201 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.888185 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q864t\"" Nov 02 11:55:44.888350 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.888216 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 11:55:44.888350 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.888240 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 11:55:44.888350 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.888252 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 11:55:44.936366 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.936318 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 11:55:44.936617 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.936601 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 11:55:44.936664 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.936639 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5rq6x" Nov 02 11:55:44.979727 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979699 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-netns\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.979815 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979733 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-multus-certs\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.979815 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:44.979815 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979770 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-netns\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.979815 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979777 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-config\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.979815 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979801 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-multus-certs\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovn-node-metrics-cert\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979866 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-socket-dir-parent\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:44.979870 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979892 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6533ee11-4dc9-4c0f-b035-1200f74d869b-host-slash\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:44.979922 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret podName:837a6c92-ea3e-490a-810e-05e2fc91a89a nodeName:}" failed. No retries permitted until 2026-11-02 11:55:45.479904056 +0000 UTC m=+3.158384497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret") pod "global-pull-secret-syncer-d5dpl" (UID: "837a6c92-ea3e-490a-810e-05e2fc91a89a") : object "kube-system"/"original-pull-secret" not registered Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979935 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6533ee11-4dc9-4c0f-b035-1200f74d869b-host-slash\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979955 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-modprobe-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979968 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-socket-dir-parent\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.979982 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-host\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980005 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-system-cni-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980008 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-host\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.980051 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980028 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cnibin\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980060 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-modprobe-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980072 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-system-cni-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8686b\" (UniqueName: \"kubernetes.io/projected/05fb7da7-c7b3-412e-a14e-45d7c10d4216-kube-api-access-8686b\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980112 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980123 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cnibin\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980140 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbbqq\" (UniqueName: \"kubernetes.io/projected/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-kube-api-access-jbbqq\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980164 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-k8s-cni-cncf-io\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhq46\" (UniqueName: \"kubernetes.io/projected/6533ee11-4dc9-4c0f-b035-1200f74d869b-kube-api-access-hhq46\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980176 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980222 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysconfig\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-systemd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-log-socket\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-system-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980300 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980317 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.980649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980347 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-hostroot\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980358 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysconfig\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980371 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bctf\" (UniqueName: \"kubernetes.io/projected/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-kube-api-access-6bctf\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980384 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-systemd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980398 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-device-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980403 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-k8s-cni-cncf-io\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980415 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-config\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980426 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-var-lib-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980442 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-log-socket\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980450 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cni-binary-copy\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-daemon-config\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980505 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-konnectivity-ca\") pod \"konnectivity-agent-5pnd2\" (UID: \"fcda3ab4-1f37-49db-8eb9-2b83fe89d484\") " pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-systemd\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980582 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-system-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980594 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-etc-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980639 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-etc-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980689 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-hostroot\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.981499 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980689 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cnibin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980695 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-var-lib-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980741 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cnibin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980742 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980739 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-kubernetes\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980783 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-kubernetes\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980793 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-device-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:44.980819 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980838 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-systemd\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:44.980864 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs podName:05fb7da7-c7b3-412e-a14e-45d7c10d4216 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:45.480849547 +0000 UTC m=+3.159330005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs") pod "network-metrics-daemon-p2frl" (UID: "05fb7da7-c7b3-412e-a14e-45d7c10d4216") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-lib-modules\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980909 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-systemd-units\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980969 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-multus\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.980990 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-kubelet\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981002 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-lib-modules\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981088 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-systemd-units\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.982425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981106 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981124 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-etc-kubernetes\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981147 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-kubelet\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981179 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-etc-kubernetes\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981201 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-multus\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981203 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc7z\" (UniqueName: \"kubernetes.io/projected/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kube-api-access-8kc7z\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981252 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtqw\" (UniqueName: \"kubernetes.io/projected/7e55d04a-613c-4e3a-bcc5-32fea4e50693-kube-api-access-djtqw\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981283 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-netns\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981311 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981338 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-bin\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981364 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e55d04a-613c-4e3a-bcc5-32fea4e50693-host\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f547b23-0439-4cc3-9127-387407a34b31-hosts-file\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981438 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-netns\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f547b23-0439-4cc3-9127-387407a34b31-tmp-dir\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981467 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-konnectivity-ca\") pod \"konnectivity-agent-5pnd2\" (UID: \"fcda3ab4-1f37-49db-8eb9-2b83fe89d484\") " pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981483 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-bin\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.983024 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981413 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-conf-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981525 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e55d04a-613c-4e3a-bcc5-32fea4e50693-host\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-registration-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981539 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981566 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-var-lib-kubelet\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981590 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981619 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-conf-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981627 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cni-binary-copy\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981639 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f547b23-0439-4cc3-9127-387407a34b31-hosts-file\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981574 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981659 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-registration-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981672 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-var-lib-kubelet\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981685 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5wc6\" (UniqueName: \"kubernetes.io/projected/291cad74-3a77-43f6-abb4-d28aaa5eb20c-kube-api-access-c5wc6\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981710 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-kubelet\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981730 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-tmp\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981754 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlnz\" (UniqueName: \"kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz\") pod \"network-check-target-s2fqn\" (UID: \"44d668e0-e534-4e0b-8d32-31ead2f678a2\") " pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:44.983814 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981777 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-agent-certs\") pod \"konnectivity-agent-5pnd2\" (UID: \"fcda3ab4-1f37-49db-8eb9-2b83fe89d484\") " pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981763 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-kubelet\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981822 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75mwz\" (UniqueName: \"kubernetes.io/projected/7f547b23-0439-4cc3-9127-387407a34b31-kube-api-access-75mwz\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981848 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-slash\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981872 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-node-log\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981895 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981916 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f547b23-0439-4cc3-9127-387407a34b31-tmp-dir\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981922 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-script-lib\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981927 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-daemon-config\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981949 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981975 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-dbus\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981993 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-node-log\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-ovn\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982028 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-netd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982042 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982095 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-dbus\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982088 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-env-overrides\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984408 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982112 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.981997 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-slash\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982112 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-ovn\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-socket-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982160 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-netd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982173 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6533ee11-4dc9-4c0f-b035-1200f74d869b-iptables-alerter-script\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982224 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e55d04a-613c-4e3a-bcc5-32fea4e50693-serviceca\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-bin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982259 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-socket-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-sys-fs\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982311 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-sys-fs\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982341 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-bin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982376 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-conf\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-run\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-tuned\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqh2j\" (UniqueName: \"kubernetes.io/projected/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-kube-api-access-xqh2j\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.984981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982479 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-sys\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-kubelet-config\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982501 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-run\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982568 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-kubelet-config\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982614 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982645 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-sys\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.982645 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-conf\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.983176 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-script-lib\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.983212 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-env-overrides\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.983370 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e55d04a-613c-4e3a-bcc5-32fea4e50693-serviceca\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.983515 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6533ee11-4dc9-4c0f-b035-1200f74d869b-iptables-alerter-script\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.984651 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovn-node-metrics-cert\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.984810 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-agent-certs\") pod \"konnectivity-agent-5pnd2\" (UID: \"fcda3ab4-1f37-49db-8eb9-2b83fe89d484\") " pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.985072 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-tmp\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:44.985468 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:44.985235 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-tuned\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:45.048507 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.048295 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhq46\" (UniqueName: \"kubernetes.io/projected/6533ee11-4dc9-4c0f-b035-1200f74d869b-kube-api-access-hhq46\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:55:45.048507 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.048433 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 11:55:45.048507 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.048456 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 11:55:45.048507 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.048468 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hhlnz for pod openshift-network-diagnostics/network-check-target-s2fqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:55:45.048805 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.048516 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz podName:44d668e0-e534-4e0b-8d32-31ead2f678a2 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:45.548500125 +0000 UTC m=+3.226980572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hhlnz" (UniqueName: "kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz") pod "network-check-target-s2fqn" (UID: "44d668e0-e534-4e0b-8d32-31ead2f678a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:55:45.048805 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.048740 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kc7z\" (UniqueName: \"kubernetes.io/projected/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kube-api-access-8kc7z\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:45.048957 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.048766 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8686b\" (UniqueName: \"kubernetes.io/projected/05fb7da7-c7b3-412e-a14e-45d7c10d4216-kube-api-access-8686b\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:45.049017 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.048988 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtqw\" (UniqueName: \"kubernetes.io/projected/7e55d04a-613c-4e3a-bcc5-32fea4e50693-kube-api-access-djtqw\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:55:45.049373 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.049345 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbbqq\" (UniqueName: \"kubernetes.io/projected/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-kube-api-access-jbbqq\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:45.049708 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.049690 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bctf\" (UniqueName: \"kubernetes.io/projected/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-kube-api-access-6bctf\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:55:45.050745 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.050724 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5wc6\" (UniqueName: \"kubernetes.io/projected/291cad74-3a77-43f6-abb4-d28aaa5eb20c-kube-api-access-c5wc6\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:45.051020 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.051003 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqh2j\" (UniqueName: \"kubernetes.io/projected/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-kube-api-access-xqh2j\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:45.051059 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.051028 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75mwz\" (UniqueName: \"kubernetes.io/projected/7f547b23-0439-4cc3-9127-387407a34b31-kube-api-access-75mwz\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:55:45.085300 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.085275 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-np85r" Nov 02 11:55:45.091911 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.091888 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:55:45.095986 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:45.095966 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e55d04a_613c_4e3a_bcc5_32fea4e50693.slice/crio-c967aa12c33d48ee034395bde5e24b790035563ce2d2101c5b2798bbcc8f94ad WatchSource:0}: Error finding container c967aa12c33d48ee034395bde5e24b790035563ce2d2101c5b2798bbcc8f94ad: Status 404 returned error can't find the container with id c967aa12c33d48ee034395bde5e24b790035563ce2d2101c5b2798bbcc8f94ad Nov 02 11:55:45.112128 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.112103 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w4h5q" Nov 02 11:55:45.113792 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.113774 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:55:45.117608 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.117590 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:55:45.120030 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.120013 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:45.121267 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:45.121247 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bebca75_b1fd_4e59_9a4b_d6589df81d9b.slice/crio-95793f9e8069e19773f8c6fe37bee884bc0428fe789ee04e369af63f13649b6f WatchSource:0}: Error finding container 95793f9e8069e19773f8c6fe37bee884bc0428fe789ee04e369af63f13649b6f: Status 404 returned error can't find the container with id 95793f9e8069e19773f8c6fe37bee884bc0428fe789ee04e369af63f13649b6f Nov 02 11:55:45.125414 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:45.125352 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f547b23_0439_4cc3_9127_387407a34b31.slice/crio-d0edb3240de6ce97514a61f70b102c54d747a84e061ec920a30bf406748872fe WatchSource:0}: Error finding container d0edb3240de6ce97514a61f70b102c54d747a84e061ec920a30bf406748872fe: Status 404 returned error can't find the container with id d0edb3240de6ce97514a61f70b102c54d747a84e061ec920a30bf406748872fe Nov 02 11:55:45.127854 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.127831 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:55:45.128072 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:45.128050 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291cad74_3a77_43f6_abb4_d28aaa5eb20c.slice/crio-df67a2733206734031c5297f313c4094f46c387533c5849890aa9501dd085c4d WatchSource:0}: Error finding container df67a2733206734031c5297f313c4094f46c387533c5849890aa9501dd085c4d: Status 404 returned error can't find the container with id df67a2733206734031c5297f313c4094f46c387533c5849890aa9501dd085c4d Nov 02 11:55:45.130176 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.130157 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:55:45.131033 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:45.131009 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcda3ab4_1f37_49db_8eb9_2b83fe89d484.slice/crio-e04937d447fc02f733c0644281139b1bad1e426678ab6fb0bcfc668fc079bcc4 WatchSource:0}: Error finding container e04937d447fc02f733c0644281139b1bad1e426678ab6fb0bcfc668fc079bcc4: Status 404 returned error can't find the container with id e04937d447fc02f733c0644281139b1bad1e426678ab6fb0bcfc668fc079bcc4 Nov 02 11:55:45.139845 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:45.139825 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcac010_ce3f_4fac_a5a0_5833466e4bd8.slice/crio-6e9fecd00c843db3b4d4eb945c394104d4467f12b2259104065c7aac804e6f35 WatchSource:0}: Error finding container 6e9fecd00c843db3b4d4eb945c394104d4467f12b2259104065c7aac804e6f35: Status 404 returned error can't find the container with id 6e9fecd00c843db3b4d4eb945c394104d4467f12b2259104065c7aac804e6f35 Nov 02 11:55:45.140689 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:55:45.140667 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6533ee11_4dc9_4c0f_b035_1200f74d869b.slice/crio-4b44493826af1f4208087805fe2e5f3256ebbf4dda2a7b23f0949a1c8469c4ad WatchSource:0}: Error finding container 4b44493826af1f4208087805fe2e5f3256ebbf4dda2a7b23f0949a1c8469c4ad: Status 404 returned error can't find the container with id 4b44493826af1f4208087805fe2e5f3256ebbf4dda2a7b23f0949a1c8469c4ad Nov 02 11:55:45.150459 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.150443 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:55:45.486239 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.486206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:45.486414 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.486254 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:45.486414 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.486366 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:55:45.486414 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.486413 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs podName:05fb7da7-c7b3-412e-a14e-45d7c10d4216 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:46.48639697 +0000 UTC m=+4.164877411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs") pod "network-metrics-daemon-p2frl" (UID: "05fb7da7-c7b3-412e-a14e-45d7c10d4216") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:55:45.486555 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.486458 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 11:55:45.486555 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.486491 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret podName:837a6c92-ea3e-490a-810e-05e2fc91a89a nodeName:}" failed. No retries permitted until 2026-11-02 11:55:46.48647986 +0000 UTC m=+4.164960294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret") pod "global-pull-secret-syncer-d5dpl" (UID: "837a6c92-ea3e-490a-810e-05e2fc91a89a") : object "kube-system"/"original-pull-secret" not registered Nov 02 11:55:45.586584 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.586552 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlnz\" (UniqueName: \"kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz\") pod \"network-check-target-s2fqn\" (UID: \"44d668e0-e534-4e0b-8d32-31ead2f678a2\") " pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:45.586734 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.586696 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 11:55:45.586734 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.586710 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 11:55:45.586734 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.586719 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hhlnz for pod openshift-network-diagnostics/network-check-target-s2fqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:55:45.586914 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:45.586780 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz podName:44d668e0-e534-4e0b-8d32-31ead2f678a2 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:46.586764943 +0000 UTC m=+4.265245370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhlnz" (UniqueName: "kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz") pod "network-check-target-s2fqn" (UID: "44d668e0-e534-4e0b-8d32-31ead2f678a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:55:45.739358 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.739272 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" event={"ID":"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b","Type":"ContainerStarted","Data":"93e69946e799b02f460aea2f6992a589061b6194070620ba185c1e0131baaa74"} Nov 02 11:55:45.741706 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.741541 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-np85r" event={"ID":"7e55d04a-613c-4e3a-bcc5-32fea4e50693","Type":"ContainerStarted","Data":"c967aa12c33d48ee034395bde5e24b790035563ce2d2101c5b2798bbcc8f94ad"} Nov 02 11:55:45.742936 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.742907 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4ncdp" event={"ID":"6533ee11-4dc9-4c0f-b035-1200f74d869b","Type":"ContainerStarted","Data":"4b44493826af1f4208087805fe2e5f3256ebbf4dda2a7b23f0949a1c8469c4ad"} Nov 02 11:55:45.744455 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.744424 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" event={"ID":"1bcac010-ce3f-4fac-a5a0-5833466e4bd8","Type":"ContainerStarted","Data":"6e9fecd00c843db3b4d4eb945c394104d4467f12b2259104065c7aac804e6f35"} Nov 02 11:55:45.745682 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.745655 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5pnd2" event={"ID":"fcda3ab4-1f37-49db-8eb9-2b83fe89d484","Type":"ContainerStarted","Data":"e04937d447fc02f733c0644281139b1bad1e426678ab6fb0bcfc668fc079bcc4"} Nov 02 11:55:45.747010 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.746989 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" event={"ID":"291cad74-3a77-43f6-abb4-d28aaa5eb20c","Type":"ContainerStarted","Data":"df67a2733206734031c5297f313c4094f46c387533c5849890aa9501dd085c4d"} Nov 02 11:55:45.748405 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.748376 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nnpfg" event={"ID":"7f547b23-0439-4cc3-9127-387407a34b31","Type":"ContainerStarted","Data":"d0edb3240de6ce97514a61f70b102c54d747a84e061ec920a30bf406748872fe"} Nov 02 11:55:45.749819 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.749792 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" event={"ID":"1c5d3609-6aef-4c1f-a876-6cb15457cc9a","Type":"ContainerStarted","Data":"ee6575e8525fa55b0a75ff3404575760b419430d526880e0722fe793bfd998ac"} Nov 02 11:55:45.751156 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.751137 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w4h5q" event={"ID":"5bebca75-b1fd-4e59-9a4b-d6589df81d9b","Type":"ContainerStarted","Data":"95793f9e8069e19773f8c6fe37bee884bc0428fe789ee04e369af63f13649b6f"} Nov 02 11:55:45.940392 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.940342 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 11:50:44 +0000 UTC" deadline="2028-07-12 04:26:54.070440186 +0000 UTC" Nov 02 11:55:45.940392 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:45.940387 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14824h31m8.130057159s" Nov 02 11:55:46.496606 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:46.494997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:46.496606 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:46.495083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:46.496606 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:46.495176 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:55:46.496606 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:46.495190 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 11:55:46.496606 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:46.495244 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs podName:05fb7da7-c7b3-412e-a14e-45d7c10d4216 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:48.495224001 +0000 UTC m=+6.173704435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs") pod "network-metrics-daemon-p2frl" (UID: "05fb7da7-c7b3-412e-a14e-45d7c10d4216") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:55:46.496606 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:46.495263 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret podName:837a6c92-ea3e-490a-810e-05e2fc91a89a nodeName:}" failed. No retries permitted until 2026-11-02 11:55:48.495253291 +0000 UTC m=+6.173733723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret") pod "global-pull-secret-syncer-d5dpl" (UID: "837a6c92-ea3e-490a-810e-05e2fc91a89a") : object "kube-system"/"original-pull-secret" not registered Nov 02 11:55:46.595984 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:46.595954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlnz\" (UniqueName: \"kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz\") pod \"network-check-target-s2fqn\" (UID: \"44d668e0-e534-4e0b-8d32-31ead2f678a2\") " pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:46.596177 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:46.596159 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 11:55:46.596259 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:46.596184 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 11:55:46.596259 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:46.596197 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hhlnz for pod openshift-network-diagnostics/network-check-target-s2fqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:55:46.596259 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:46.596251 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz podName:44d668e0-e534-4e0b-8d32-31ead2f678a2 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:48.596233265 +0000 UTC m=+6.274713727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhlnz" (UniqueName: "kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz") pod "network-check-target-s2fqn" (UID: "44d668e0-e534-4e0b-8d32-31ead2f678a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:55:46.732842 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:46.732223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:46.732842 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:46.732365 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:55:46.732842 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:46.732457 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:46.732842 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:46.732548 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:55:46.732842 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:46.732641 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:46.732842 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:46.732763 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:55:46.941589 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:46.940670 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 11:50:44 +0000 UTC" deadline="2028-08-09 17:19:27.291576004 +0000 UTC" Nov 02 11:55:46.941589 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:46.940738 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15509h23m40.350842653s" Nov 02 11:55:47.768273 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:47.768058 2582 generic.go:356] "Generic (PLEG): container finished" podID="14b5e76aab2a7b6af2007243418d856f" containerID="50657f8c433699736a7ec72876c559843cb98968d8f8e868fc90a9d9415d51e0" exitCode=0 Nov 02 11:55:47.768821 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:47.768544 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" event={"ID":"14b5e76aab2a7b6af2007243418d856f","Type":"ContainerDied","Data":"50657f8c433699736a7ec72876c559843cb98968d8f8e868fc90a9d9415d51e0"} Nov 02 11:55:47.775759 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:47.775323 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" event={"ID":"1ebeef05fd06ac44ab33873433450b74","Type":"ContainerStarted","Data":"514cbe11b9da4fe6a7927abdfa89423d2ad8ab9b88833f7fa1047c56af4e4e42"} Nov 02 11:55:48.514169 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:48.513402 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:48.514169 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:48.513517 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:48.514169 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:48.513647 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 11:55:48.514169 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:48.513705 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret podName:837a6c92-ea3e-490a-810e-05e2fc91a89a nodeName:}" failed. No retries permitted until 2026-11-02 11:55:52.513686453 +0000 UTC m=+10.192166885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret") pod "global-pull-secret-syncer-d5dpl" (UID: "837a6c92-ea3e-490a-810e-05e2fc91a89a") : object "kube-system"/"original-pull-secret" not registered Nov 02 11:55:48.514169 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:48.514077 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:55:48.514169 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:48.514137 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs podName:05fb7da7-c7b3-412e-a14e-45d7c10d4216 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:52.514120024 +0000 UTC m=+10.192600452 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs") pod "network-metrics-daemon-p2frl" (UID: "05fb7da7-c7b3-412e-a14e-45d7c10d4216") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:55:48.614222 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:48.614192 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlnz\" (UniqueName: \"kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz\") pod \"network-check-target-s2fqn\" (UID: \"44d668e0-e534-4e0b-8d32-31ead2f678a2\") " pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:48.614354 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:48.614336 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 11:55:48.614421 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:48.614358 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 11:55:48.614421 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:48.614371 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hhlnz for pod openshift-network-diagnostics/network-check-target-s2fqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:55:48.614535 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:48.614422 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz podName:44d668e0-e534-4e0b-8d32-31ead2f678a2 nodeName:}" failed. No retries permitted until 2026-11-02 11:55:52.614405482 +0000 UTC m=+10.292885925 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhlnz" (UniqueName: "kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz") pod "network-check-target-s2fqn" (UID: "44d668e0-e534-4e0b-8d32-31ead2f678a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:55:48.732021 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:48.731543 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:48.732021 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:48.731659 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:55:48.732021 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:48.731725 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:48.732021 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:48.731804 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:55:48.732021 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:48.731857 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:48.732021 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:48.731919 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:55:48.789942 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:48.789189 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" event={"ID":"14b5e76aab2a7b6af2007243418d856f","Type":"ContainerStarted","Data":"86c19a00a41adfe03b0d6a13adbc6aa920acb1a05b636c69557525f035a6ea8b"} Nov 02 11:55:48.804657 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:48.804586 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" event={"ID":"291cad74-3a77-43f6-abb4-d28aaa5eb20c","Type":"ContainerStarted","Data":"0186e52975365ac8864dcad8376abc551977143341a77a14897c7f0944467e7f"} Nov 02 11:55:48.908062 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:48.907470 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" podStartSLOduration=4.907458211 podStartE2EDuration="4.907458211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:48.90696171 +0000 UTC m=+6.585442159" watchObservedRunningTime="2026-11-02 11:55:48.907458211 +0000 UTC m=+6.585938659" Nov 02 11:55:48.908062 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:48.907625 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" podStartSLOduration=4.907620168 podStartE2EDuration="4.907620168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:48.006450845 +0000 UTC m=+5.684931294" watchObservedRunningTime="2026-11-02 11:55:48.907620168 +0000 UTC m=+6.586100616" Nov 02 11:55:49.826285 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:49.826250 2582 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="0186e52975365ac8864dcad8376abc551977143341a77a14897c7f0944467e7f" exitCode=0 Nov 02 11:55:49.826714 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:49.826628 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" event={"ID":"291cad74-3a77-43f6-abb4-d28aaa5eb20c","Type":"ContainerDied","Data":"0186e52975365ac8864dcad8376abc551977143341a77a14897c7f0944467e7f"} Nov 02 11:55:50.732902 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:50.732151 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:50.732902 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:50.732279 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:55:50.732902 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:50.732577 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:50.732902 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:50.732663 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:55:50.732902 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:50.732710 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:50.732902 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:50.732788 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:55:50.831600 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:50.830943 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5pnd2" event={"ID":"fcda3ab4-1f37-49db-8eb9-2b83fe89d484","Type":"ContainerStarted","Data":"8662465156ef5ab179c3a664575ddfa83c3c0a5018840fa813e60583b19912a0"} Nov 02 11:55:50.836132 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:50.835489 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-np85r" event={"ID":"7e55d04a-613c-4e3a-bcc5-32fea4e50693","Type":"ContainerStarted","Data":"6cd1a0fb5339fffd0c9a70b4b2929efe6a51fc8dd8a287d3c89a3ebf2a2344f9"} Nov 02 11:55:51.066168 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:51.064851 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5pnd2" podStartSLOduration=3.445829082 podStartE2EDuration="8.064831068s" totalImagesPullingTime="4.619001986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:50.949177036 +0000 UTC m=+8.627657486" watchObservedRunningTime="2026-11-02 11:55:51.064831068 +0000 UTC m=+8.743311516" Nov 02 11:55:51.841651 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:51.841614 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" event={"ID":"1bcac010-ce3f-4fac-a5a0-5833466e4bd8","Type":"ContainerStarted","Data":"02738323e583ed88350e5a3685b2425bac0d5a76885b8b8e2d1f838d81bc775c"} Nov 02 11:55:52.545231 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:52.545195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:52.545430 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:52.545255 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:52.545430 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:52.545369 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:55:52.545577 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:52.545442 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs podName:05fb7da7-c7b3-412e-a14e-45d7c10d4216 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:00.545423985 +0000 UTC m=+18.223904413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs") pod "network-metrics-daemon-p2frl" (UID: "05fb7da7-c7b3-412e-a14e-45d7c10d4216") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:55:52.545879 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:52.545851 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 11:55:52.545978 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:52.545910 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret podName:837a6c92-ea3e-490a-810e-05e2fc91a89a nodeName:}" failed. No retries permitted until 2026-11-02 11:56:00.545895758 +0000 UTC m=+18.224376190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret") pod "global-pull-secret-syncer-d5dpl" (UID: "837a6c92-ea3e-490a-810e-05e2fc91a89a") : object "kube-system"/"original-pull-secret" not registered Nov 02 11:55:52.645876 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:52.645842 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlnz\" (UniqueName: \"kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz\") pod \"network-check-target-s2fqn\" (UID: \"44d668e0-e534-4e0b-8d32-31ead2f678a2\") " pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:52.646020 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:52.646008 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 11:55:52.646070 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:52.646024 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 11:55:52.646070 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:52.646033 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hhlnz for pod openshift-network-diagnostics/network-check-target-s2fqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:55:52.646127 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:52.646076 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz podName:44d668e0-e534-4e0b-8d32-31ead2f678a2 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:00.646062817 +0000 UTC m=+18.324543257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhlnz" (UniqueName: "kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz") pod "network-check-target-s2fqn" (UID: "44d668e0-e534-4e0b-8d32-31ead2f678a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:55:52.732372 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:52.732343 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:52.732544 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:52.732467 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:55:52.732629 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:52.732608 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:52.732719 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:52.732699 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:55:52.732781 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:52.732770 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:52.732881 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:52.732856 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:55:53.697981 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:53.697956 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 11:55:53.749782 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:53.749646 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T11:55:53.697976302Z","UUID":"ee68eb37-6a43-4d8a-a9c6-eac9f7d263ad","Handler":null,"Name":"","Endpoint":""} Nov 02 11:55:53.751215 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:53.751197 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 11:55:53.751289 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:53.751227 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 11:55:53.847059 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:53.846996 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nnpfg" event={"ID":"7f547b23-0439-4cc3-9127-387407a34b31","Type":"ContainerStarted","Data":"4e15fa368a74c75b00398345bc56b972c3754f0715bd50e10f1f58711b68ddad"} Nov 02 11:55:53.850099 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:53.850071 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w4h5q" event={"ID":"5bebca75-b1fd-4e59-9a4b-d6589df81d9b","Type":"ContainerStarted","Data":"1e090bfc2b54c4e384ebd06504d7f71ea031928956058837cda38a8cb9346de3"} Nov 02 11:55:53.852724 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:53.852698 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" event={"ID":"1bcac010-ce3f-4fac-a5a0-5833466e4bd8","Type":"ContainerStarted","Data":"9b33b697f2501333abb54b7467cced6fb9f90e4b2aea235c094decd7b5f13e75"} Nov 02 11:55:53.963046 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:53.962805 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-np85r" podStartSLOduration=5.44654492 podStartE2EDuration="10.962787261s" totalImagesPullingTime="5.516242341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:51.066109321 +0000 UTC m=+8.744589773" watchObservedRunningTime="2026-11-02 11:55:53.962787261 +0000 UTC m=+11.641267710" Nov 02 11:55:53.967062 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:53.966918 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-nnpfg" podStartSLOduration=2.774406856 podStartE2EDuration="10.966895547s" totalImagesPullingTime="8.192488691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:53.963073599 +0000 UTC m=+11.641554050" watchObservedRunningTime="2026-11-02 11:55:53.966895547 +0000 UTC m=+11.645375992" Nov 02 11:55:54.016881 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:54.016847 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:54.017431 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:54.017414 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:54.083428 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:54.083147 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-w4h5q" podStartSLOduration=2.889325468 podStartE2EDuration="11.083130098s" totalImagesPullingTime="8.19380463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:54.082334456 +0000 UTC m=+11.760814906" watchObservedRunningTime="2026-11-02 11:55:54.083130098 +0000 UTC m=+11.761610724" Nov 02 11:55:54.730619 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:54.730585 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:54.730619 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:54.730603 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:54.731049 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:54.730764 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:55:54.731049 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:54.730815 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:55:54.731049 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:54.730861 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:54.731049 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:54.730962 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:55:54.858140 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:54.858105 2582 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="38fc2a3de5db76b3cef5f263bc10bb2d01f0e24423b2ba1e1282f3c05c580f41" exitCode=0 Nov 02 11:55:54.859268 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:54.859243 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" event={"ID":"291cad74-3a77-43f6-abb4-d28aaa5eb20c","Type":"ContainerDied","Data":"38fc2a3de5db76b3cef5f263bc10bb2d01f0e24423b2ba1e1282f3c05c580f41"} Nov 02 11:55:54.859372 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:54.859288 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:54.860812 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:54.860795 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:55:55.861652 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:55.861619 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4ncdp" event={"ID":"6533ee11-4dc9-4c0f-b035-1200f74d869b","Type":"ContainerStarted","Data":"2fd35d12beb0d82d96633c2d623b966f29dd75efd0640d1defb6c53a738e7805"} Nov 02 11:55:55.865278 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:55.865251 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" event={"ID":"1bcac010-ce3f-4fac-a5a0-5833466e4bd8","Type":"ContainerStarted","Data":"419e08e5744c3e007ec1d5e232a6f67d55fbf1a941b8baaebb95e78ec28061fb"} Nov 02 11:55:55.867125 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:55.867084 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" event={"ID":"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b","Type":"ContainerStarted","Data":"3a4341a4c01f9923e9086bd397dee2e358d56f33109fd41540271fa6100ecdf4"} Nov 02 11:55:55.978196 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:55.978103 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4ncdp" podStartSLOduration=4.529583472 podStartE2EDuration="12.978089241s" totalImagesPullingTime="8.448505769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:55.977475853 +0000 UTC m=+13.655956301" watchObservedRunningTime="2026-11-02 11:55:55.978089241 +0000 UTC m=+13.656569688" Nov 02 11:55:56.098264 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:56.098217 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" podStartSLOduration=3.390047266 podStartE2EDuration="13.098201424s" totalImagesPullingTime="9.708154158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:56.098033475 +0000 UTC m=+13.776513924" watchObservedRunningTime="2026-11-02 11:55:56.098201424 +0000 UTC m=+13.776681871" Nov 02 11:55:56.217777 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:56.217733 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" podStartSLOduration=2.678657293 podStartE2EDuration="13.217721253s" totalImagesPullingTime="10.53906396s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:55:56.217568486 +0000 UTC m=+13.896048933" watchObservedRunningTime="2026-11-02 11:55:56.217721253 +0000 UTC m=+13.896201691" Nov 02 11:55:56.730151 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:56.730096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:56.730151 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:56.730135 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:56.730398 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:56.730204 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:56.730398 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:56.730313 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:55:56.730398 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:56.730378 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:55:56.730553 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:56.730460 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:55:56.871527 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:56.871480 2582 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="6627f0fa0008579dbbcf9caf6198766919d5cbec806bdc29a1d74af7e0dff6b1" exitCode=0 Nov 02 11:55:56.871988 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:56.871529 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" event={"ID":"291cad74-3a77-43f6-abb4-d28aaa5eb20c","Type":"ContainerDied","Data":"6627f0fa0008579dbbcf9caf6198766919d5cbec806bdc29a1d74af7e0dff6b1"} Nov 02 11:55:58.729425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:58.729393 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:55:58.729425 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:58.729430 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:55:58.730067 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:58.729393 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:55:58.730067 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:58.729574 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:55:58.730067 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:58.729661 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:55:58.730067 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:55:58.729743 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:55:58.878016 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:58.877982 2582 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="a6b506fb2ffc5103610d4e87908d89491eee01c95b9a0797eee060f8a635d780" exitCode=0 Nov 02 11:55:58.878179 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:55:58.878028 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" event={"ID":"291cad74-3a77-43f6-abb4-d28aaa5eb20c","Type":"ContainerDied","Data":"a6b506fb2ffc5103610d4e87908d89491eee01c95b9a0797eee060f8a635d780"} Nov 02 11:56:00.607085 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.606705 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:00.607534 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.607094 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:00.607534 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:00.606820 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:56:00.607534 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:00.607165 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs podName:05fb7da7-c7b3-412e-a14e-45d7c10d4216 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:16.607149147 +0000 UTC m=+34.285629578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs") pod "network-metrics-daemon-p2frl" (UID: "05fb7da7-c7b3-412e-a14e-45d7c10d4216") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 11:56:00.607534 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:00.607185 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 11:56:00.607534 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:00.607229 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret podName:837a6c92-ea3e-490a-810e-05e2fc91a89a nodeName:}" failed. No retries permitted until 2026-11-02 11:56:16.607216065 +0000 UTC m=+34.285696492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret") pod "global-pull-secret-syncer-d5dpl" (UID: "837a6c92-ea3e-490a-810e-05e2fc91a89a") : object "kube-system"/"original-pull-secret" not registered Nov 02 11:56:00.707819 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.707789 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlnz\" (UniqueName: \"kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz\") pod \"network-check-target-s2fqn\" (UID: \"44d668e0-e534-4e0b-8d32-31ead2f678a2\") " pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:00.707957 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:00.707898 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 11:56:00.707957 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:00.707916 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 11:56:00.707957 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:00.707928 2582 projected.go:196] Error preparing data for projected volume kube-api-access-hhlnz for pod openshift-network-diagnostics/network-check-target-s2fqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:56:00.708075 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:00.707977 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz podName:44d668e0-e534-4e0b-8d32-31ead2f678a2 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:16.707960869 +0000 UTC m=+34.386441312 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hhlnz" (UniqueName: "kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz") pod "network-check-target-s2fqn" (UID: "44d668e0-e534-4e0b-8d32-31ead2f678a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 11:56:00.729989 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.729967 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:00.730100 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:00.730059 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:56:00.730100 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.729965 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:00.730100 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.730090 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:00.730220 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:00.730160 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:56:00.730294 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:00.730270 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:56:00.884548 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.884518 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" event={"ID":"1c5d3609-6aef-4c1f-a876-6cb15457cc9a","Type":"ContainerStarted","Data":"2e34cefbe72e55756efbad855781412da7aa1dc313ace8b640b114212e34b8b1"} Nov 02 11:56:00.884649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.884580 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" event={"ID":"1c5d3609-6aef-4c1f-a876-6cb15457cc9a","Type":"ContainerStarted","Data":"42f4d42f8313f422c6c0465bbda7c3907334e6ba44b29d7ccd62e34a09735e16"} Nov 02 11:56:00.884649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.884610 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" event={"ID":"1c5d3609-6aef-4c1f-a876-6cb15457cc9a","Type":"ContainerStarted","Data":"57f863895cc71091c31329127f05202d77ca37d82b0ba2065c79235b0d877c94"} Nov 02 11:56:00.884649 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.884639 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" event={"ID":"1c5d3609-6aef-4c1f-a876-6cb15457cc9a","Type":"ContainerStarted","Data":"e1609ae78196c5a48029a1da61a5eef52adb2d93752bc96be06ee89b82afc4f4"} Nov 02 11:56:00.884852 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.884666 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" event={"ID":"1c5d3609-6aef-4c1f-a876-6cb15457cc9a","Type":"ContainerStarted","Data":"d0cd377f55d90bd9a086932fa76099a97649ee7d1dfc6d313bbe038322281441"} Nov 02 11:56:00.884852 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:00.884693 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" event={"ID":"1c5d3609-6aef-4c1f-a876-6cb15457cc9a","Type":"ContainerStarted","Data":"9f1eb2eb662c5b0a56030a29d4a717bdadb8fc0a250c4ffe31d723290ed9d193"} Nov 02 11:56:01.891913 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:01.891875 2582 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="ffb22e091dbc5b383057a78c1a867dd96cc382a22ef3fe7656fd6326a9f85ff8" exitCode=0 Nov 02 11:56:01.892390 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:01.891952 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" event={"ID":"291cad74-3a77-43f6-abb4-d28aaa5eb20c","Type":"ContainerDied","Data":"ffb22e091dbc5b383057a78c1a867dd96cc382a22ef3fe7656fd6326a9f85ff8"} Nov 02 11:56:02.297186 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:02.297157 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:02.604449 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:02.604382 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:02.729995 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:02.729967 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:02.730097 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:02.730075 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:02.730138 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:02.730088 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:56:02.730176 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:02.730141 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:02.730278 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:02.730258 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:56:02.730367 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:02.730348 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:56:02.849014 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:02.848986 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:02.896529 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:02.896456 2582 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="e231929b133c224219969be6f8d8e0a64605eabcd252a48113eeb8c174c091df" exitCode=0 Nov 02 11:56:02.897026 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:02.896544 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" event={"ID":"291cad74-3a77-43f6-abb4-d28aaa5eb20c","Type":"ContainerDied","Data":"e231929b133c224219969be6f8d8e0a64605eabcd252a48113eeb8c174c091df"} Nov 02 11:56:02.900466 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:02.900440 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" event={"ID":"1c5d3609-6aef-4c1f-a876-6cb15457cc9a","Type":"ContainerStarted","Data":"a48d1b24259c5de6fceb20fce878745f0d87b6eeddd45a27edf23f01bcd15739"} Nov 02 11:56:03.906984 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:03.906950 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" event={"ID":"291cad74-3a77-43f6-abb4-d28aaa5eb20c","Type":"ContainerStarted","Data":"a7c926eae5a72c4731665282c99eb4389cc03312ded08ec7a95bff053264f5ae"} Nov 02 11:56:04.030998 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:04.030955 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mmzdg" podStartSLOduration=8.864328989 podStartE2EDuration="21.03094321s" totalImagesPullingTime="12.166614221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:04.030489539 +0000 UTC m=+21.708969988" watchObservedRunningTime="2026-11-02 11:56:04.03094321 +0000 UTC m=+21.709423655" Nov 02 11:56:04.729161 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:04.729131 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:04.729307 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:04.729138 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:04.729307 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:04.729244 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:56:04.729307 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:04.729252 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:04.729535 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:04.729339 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:56:04.729535 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:04.729439 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:56:04.916285 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:04.916252 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" event={"ID":"1c5d3609-6aef-4c1f-a876-6cb15457cc9a","Type":"ContainerStarted","Data":"31807dbc0076f377b2c8717ffdb081badcf7cd514301091f0b6f1a618bad5858"} Nov 02 11:56:05.200611 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:05.200543 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" podStartSLOduration=7.332007714 podStartE2EDuration="22.200531638s" totalImagesPullingTime="14.868523924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:05.149513262 +0000 UTC m=+22.827993707" watchObservedRunningTime="2026-11-02 11:56:05.200531638 +0000 UTC m=+22.879012088" Nov 02 11:56:05.918286 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:05.918259 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 11:56:05.918715 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:05.918673 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:05.918769 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:05.918736 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:05.933007 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:05.932985 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:05.933173 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:05.933158 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:05.942135 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:05.942113 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:06.223419 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:06.223347 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:06.730001 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:06.729975 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:06.730162 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:06.730010 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:06.730162 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:06.730081 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:56:06.730162 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:06.730149 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:06.730317 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:06.730258 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:56:06.730367 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:06.730341 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:56:06.891754 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:06.891719 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:06.920748 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:06.920729 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 11:56:07.137907 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:07.137846 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:07.233126 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:07.233098 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d5dpl"] Nov 02 11:56:07.233295 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:07.233206 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:07.233361 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:07.233323 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:56:07.233905 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:07.233879 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p2frl"] Nov 02 11:56:07.234007 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:07.233954 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:07.234050 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:07.234025 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:56:07.235578 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:07.235556 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s2fqn"] Nov 02 11:56:07.235694 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:07.235640 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:07.235754 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:07.235731 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:56:07.922518 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:07.922476 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 11:56:08.729217 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:08.729190 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:08.729358 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:08.729191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:08.729358 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:08.729191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:08.729358 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:08.729308 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:56:08.729487 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:08.729375 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:56:08.729487 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:08.729450 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:56:09.293914 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:09.293887 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:09.294290 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:09.294152 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 11:56:09.304222 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:09.304190 2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" podUID="1c5d3609-6aef-4c1f-a876-6cb15457cc9a" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 11:56:09.312347 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:09.312319 2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" podUID="1c5d3609-6aef-4c1f-a876-6cb15457cc9a" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 11:56:10.203171 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:10.203141 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:10.458681 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:10.458615 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:10.707069 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:10.707043 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:10.729139 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:10.729077 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:10.729139 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:10.729109 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:10.729281 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:10.729164 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:56:10.729281 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:10.729240 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:56:10.729385 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:10.729311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:10.729491 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:10.729472 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:56:10.951802 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:10.951779 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:11.915216 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:11.915181 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:12.190781 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:12.190532 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:12.729881 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:12.729853 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:12.730022 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:12.729949 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p2frl" podUID="05fb7da7-c7b3-412e-a14e-45d7c10d4216" Nov 02 11:56:12.730094 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:12.730037 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:12.730156 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:12.730143 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:12.730230 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:12.730202 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5dpl" podUID="837a6c92-ea3e-490a-810e-05e2fc91a89a" Nov 02 11:56:12.730282 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:12.730224 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s2fqn" podUID="44d668e0-e534-4e0b-8d32-31ead2f678a2" Nov 02 11:56:12.774436 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:12.774413 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:13.163747 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:13.163722 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeReady" Nov 02 11:56:13.164039 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:13.163815 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 11:56:13.305121 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:13.305071 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:13.547489 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:13.547459 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:14.729446 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:14.729411 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:14.729446 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:14.729439 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:14.729982 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:14.729553 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:14.785459 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:14.785435 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:14.796808 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:14.796784 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 11:56:14.796924 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:14.796912 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 11:56:14.848172 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:14.848153 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 11:56:14.848238 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:14.848185 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lk8z8\"" Nov 02 11:56:14.848238 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:14.848185 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hc67r\"" Nov 02 11:56:14.848300 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:14.848238 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 11:56:14.848300 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:14.848298 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gfx2g\"" Nov 02 11:56:15.996623 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:15.996586 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:16.616781 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:16.616757 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:16.616866 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:16.616796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:16.616915 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:16.616901 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 11:56:16.616971 ip-10-0-141-170 kubenswrapper[2582]: E1102 11:56:16.616961 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs podName:05fb7da7-c7b3-412e-a14e-45d7c10d4216 nodeName:}" failed. No retries permitted until 2026-11-02 11:56:48.616945535 +0000 UTC m=+66.295425961 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs") pod "network-metrics-daemon-p2frl" (UID: "05fb7da7-c7b3-412e-a14e-45d7c10d4216") : secret "metrics-daemon-secret" not found Nov 02 11:56:16.621122 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:16.621090 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:16.717305 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:16.717280 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlnz\" (UniqueName: \"kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz\") pod \"network-check-target-s2fqn\" (UID: \"44d668e0-e534-4e0b-8d32-31ead2f678a2\") " pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:16.719445 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:16.719417 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlnz\" (UniqueName: \"kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz\") pod \"network-check-target-s2fqn\" (UID: \"44d668e0-e534-4e0b-8d32-31ead2f678a2\") " pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:16.847883 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:16.847859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:16.851979 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:16.851963 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:17.231328 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:17.231296 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d5dpl"] Nov 02 11:56:17.235011 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:56:17.234985 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837a6c92_ea3e_490a_810e_05e2fc91a89a.slice/crio-c64235b0453ba3de6be7227ff41aa509ff11973ffa307f1a461fbc048d322e11 WatchSource:0}: Error finding container c64235b0453ba3de6be7227ff41aa509ff11973ffa307f1a461fbc048d322e11: Status 404 returned error can't find the container with id c64235b0453ba3de6be7227ff41aa509ff11973ffa307f1a461fbc048d322e11 Nov 02 11:56:17.280322 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:17.280304 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s2fqn"] Nov 02 11:56:17.282361 ip-10-0-141-170 kubenswrapper[2582]: W1102 11:56:17.282338 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d668e0_e534_4e0b_8d32_31ead2f678a2.slice/crio-18f2d976231ccb21e879c9163ef4ccb85dee26fb1a603689982c91d2adffeff9 WatchSource:0}: Error finding container 18f2d976231ccb21e879c9163ef4ccb85dee26fb1a603689982c91d2adffeff9: Status 404 returned error can't find the container with id 18f2d976231ccb21e879c9163ef4ccb85dee26fb1a603689982c91d2adffeff9 Nov 02 11:56:17.945806 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:17.945770 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d5dpl" event={"ID":"837a6c92-ea3e-490a-810e-05e2fc91a89a","Type":"ContainerStarted","Data":"c64235b0453ba3de6be7227ff41aa509ff11973ffa307f1a461fbc048d322e11"} Nov 02 11:56:17.946898 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:17.946865 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s2fqn" event={"ID":"44d668e0-e534-4e0b-8d32-31ead2f678a2","Type":"ContainerStarted","Data":"18f2d976231ccb21e879c9163ef4ccb85dee26fb1a603689982c91d2adffeff9"} Nov 02 11:56:17.999325 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:17.999304 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:18.250125 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:18.250061 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:18.499437 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:18.499401 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:18.745373 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:18.745328 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:19.010317 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:19.010175 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:19.272539 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:19.272459 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nnpfg_7f547b23-0439-4cc3-9127-387407a34b31/dns-node-resolver/0.log" Nov 02 11:56:19.863688 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:19.863654 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:20.075526 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:20.075484 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-np85r_7e55d04a-613c-4e3a-bcc5-32fea4e50693/node-ca/0.log" Nov 02 11:56:20.110652 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:20.110575 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:20.355235 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:20.355202 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:20.602323 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:20.602274 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:20.853828 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:20.853798 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:21.101866 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:21.101831 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:21.869907 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:21.869801 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:21.958806 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:21.958774 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s2fqn" event={"ID":"44d668e0-e534-4e0b-8d32-31ead2f678a2","Type":"ContainerStarted","Data":"7b0df74dc8cd7cd91fd196b3e00d23672d2a3023ca16227d30c4dfa85fd8f067"} Nov 02 11:56:21.958965 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:21.958915 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:22.077316 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:22.077272 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s2fqn" podStartSLOduration=34.849333806 podStartE2EDuration="39.077256591s" totalImagesPullingTime="4.227922785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:22.07661388 +0000 UTC m=+39.755094327" watchObservedRunningTime="2026-11-02 11:56:22.077256591 +0000 UTC m=+39.755737040" Nov 02 11:56:22.116316 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:22.116291 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:22.356524 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:22.356479 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:22.601860 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:22.601831 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:22.850843 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:22.850812 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:22.906319 ip-10-0-141-170 kubenswrapper[2582]: I1102 11:56:22.906209 2582 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 11:56:22.906394 ip-10-0-141-170 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 11:56:22.930880 ip-10-0-141-170 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 11:56:22.931059 ip-10-0-141-170 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 11:56:22.931111 ip-10-0-141-170 systemd[1]: kubelet.service: Consumed 2.102s CPU time. Nov 02 11:56:22.941945 ip-10-0-141-170 systemd[1]: Starting Kubernetes Kubelet... Nov 02 11:56:22.998507 ip-10-0-141-170 kubenswrapper[4312]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 11:56:22.998507 ip-10-0-141-170 kubenswrapper[4312]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 11:56:22.998937 ip-10-0-141-170 kubenswrapper[4312]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 11:56:22.998937 ip-10-0-141-170 kubenswrapper[4312]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 11:56:23.000451 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000386 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:56:23.000451 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000451 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000455 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000458 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000461 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000464 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000467 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000470 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000473 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000476 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000478 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000481 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000484 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000487 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000489 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000509 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000512 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000515 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000517 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000520 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:56:23.000536 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000523 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000526 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000528 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000531 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000533 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000536 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000543 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000547 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000549 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000552 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000555 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000557 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000560 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000563 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000565 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000568 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000570 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000573 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000575 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000578 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:56:23.001046 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000580 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000583 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000585 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000588 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000590 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000594 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000598 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000603 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000606 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000609 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000611 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000614 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000617 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000621 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000625 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000629 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000632 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000634 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000637 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:56:23.001582 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000640 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000642 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000645 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000647 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000650 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000653 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000655 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000658 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000661 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000663 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000666 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000668 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000671 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000673 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000676 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000678 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000681 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000684 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000686 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000689 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:56:23.002055 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000692 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000695 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000697 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000700 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000703 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000705 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000708 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000710 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000713 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000716 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000719 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000722 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000724 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000728 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000730 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000733 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000735 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000738 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000740 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:56:23.002583 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000743 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000746 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000748 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000751 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000753 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000756 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000758 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.000761 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001538 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001546 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001549 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001552 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001554 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001558 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001561 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001563 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001566 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001569 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001572 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:56:23.003053 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001575 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001578 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001580 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001584 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001586 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001589 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001592 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001595 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001598 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001600 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001603 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001606 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001608 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001611 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001613 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001616 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001619 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001621 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001624 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001626 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:56:23.003541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001629 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001631 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001634 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001636 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001639 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001642 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001644 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001648 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001650 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001653 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001656 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001659 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001661 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001664 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001667 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001669 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001672 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001674 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001677 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001679 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:56:23.004031 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001682 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001684 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001687 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001689 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001692 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001695 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001697 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001700 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001702 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001705 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001708 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001710 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001713 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001715 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001719 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001721 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001724 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001726 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001729 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:56:23.004541 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001733 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001736 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001739 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001741 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001744 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001746 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001749 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001751 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001753 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001756 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001759 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001761 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001764 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001766 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001770 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001775 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001779 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001782 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001786 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:56:23.005003 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001789 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001792 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001795 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001797 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001800 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001802 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001805 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001807 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001810 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001813 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001816 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001818 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001821 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001824 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001827 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001830 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.001832 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002300 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002305 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:56:23.005470 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002308 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002310 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002313 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002315 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002318 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002321 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002323 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002326 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002329 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002331 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002334 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002337 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002340 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002342 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002345 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002347 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002350 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002352 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002355 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002357 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:56:23.006075 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002360 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002363 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002366 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002369 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002372 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002374 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002377 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002380 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002383 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002386 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002388 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002391 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002394 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002396 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002399 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002401 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002404 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002406 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002409 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:56:23.006945 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002412 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002414 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002417 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002419 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002421 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002424 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002426 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002429 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002432 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002434 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002437 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002439 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002442 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002445 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002447 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002450 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002453 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002455 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002458 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:56:23.007783 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002460 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002463 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002465 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002468 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002471 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002474 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002476 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002478 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002481 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002484 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002487 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002490 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002506 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002509 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002513 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002517 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002521 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002525 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002529 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002532 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:56:23.008310 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002535 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002539 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002544 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002547 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002550 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002553 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002557 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002560 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002562 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002565 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002568 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002570 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002573 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002575 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002580 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002583 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002586 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002590 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002592 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:56:23.008835 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002595 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002598 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002600 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002603 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002605 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002608 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.002612 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002703 4312 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002717 4312 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002723 4312 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002728 4312 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002732 4312 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002736 4312 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002740 4312 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002749 4312 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002752 4312 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002756 4312 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002759 4312 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002763 4312 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002766 4312 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002769 4312 flags.go:64] FLAG: --cgroup-root="" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002772 4312 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 11:56:23.009302 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002775 4312 flags.go:64] FLAG: --client-ca-file="" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002778 4312 flags.go:64] FLAG: --cloud-provider="external" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002781 4312 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002785 4312 flags.go:64] FLAG: --cluster-domain="" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002788 4312 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002794 4312 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002797 4312 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002800 4312 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002805 4312 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002808 4312 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002811 4312 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002815 4312 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002818 4312 flags.go:64] FLAG: --contention-profiling="false" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002821 4312 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002824 4312 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002827 4312 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002830 4312 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002834 4312 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002837 4312 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002840 4312 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002843 4312 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002846 4312 flags.go:64] FLAG: --enable-server="true" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002849 4312 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002854 4312 flags.go:64] FLAG: --event-burst="100" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002857 4312 flags.go:64] FLAG: --event-qps="50" Nov 02 11:56:23.009857 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002860 4312 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002863 4312 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002866 4312 flags.go:64] FLAG: --eviction-hard="" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002871 4312 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002874 4312 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002877 4312 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002880 4312 flags.go:64] FLAG: --eviction-soft="" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002883 4312 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002886 4312 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002889 4312 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002892 4312 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002895 4312 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002899 4312 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002902 4312 flags.go:64] FLAG: --feature-gates="" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002906 4312 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002909 4312 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002913 4312 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002917 4312 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002920 4312 flags.go:64] FLAG: --healthz-port="10248" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002923 4312 flags.go:64] FLAG: --help="false" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002926 4312 flags.go:64] FLAG: --hostname-override="ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002929 4312 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002933 4312 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 11:56:23.010481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002936 4312 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002939 4312 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002942 4312 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002945 4312 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002948 4312 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002951 4312 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002954 4312 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002957 4312 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002960 4312 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002963 4312 flags.go:64] FLAG: --kube-reserved="" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002966 4312 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002969 4312 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002972 4312 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002975 4312 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002978 4312 flags.go:64] FLAG: --lock-file="" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002981 4312 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002984 4312 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002987 4312 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002992 4312 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002995 4312 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.002998 4312 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003002 4312 flags.go:64] FLAG: --logging-format="text" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003005 4312 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003009 4312 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 11:56:23.011061 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003012 4312 flags.go:64] FLAG: --manifest-url="" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003015 4312 flags.go:64] FLAG: --manifest-url-header="" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003020 4312 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003025 4312 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003028 4312 flags.go:64] FLAG: --max-pods="110" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003031 4312 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003034 4312 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003037 4312 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003041 4312 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003043 4312 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003047 4312 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003050 4312 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003057 4312 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003060 4312 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003063 4312 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003067 4312 flags.go:64] FLAG: --pod-cidr="" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003070 4312 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003072 4312 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003075 4312 flags.go:64] FLAG: --pods-per-core="0" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003078 4312 flags.go:64] FLAG: --port="10250" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003081 4312 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003084 4312 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b26846743edd24ea" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003088 4312 flags.go:64] FLAG: --qos-reserved="" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003091 4312 flags.go:64] FLAG: --read-only-port="10255" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003094 4312 flags.go:64] FLAG: --register-node="true" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003097 4312 flags.go:64] FLAG: --register-with-taints="" Nov 02 11:56:23.011737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003101 4312 flags.go:64] FLAG: --registry-burst="10" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003104 4312 flags.go:64] FLAG: --registry-qps="5" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003107 4312 flags.go:64] FLAG: --reserved-cpus="" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003112 4312 flags.go:64] FLAG: --reserved-memory="" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003116 4312 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003120 4312 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003123 4312 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003126 4312 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003129 4312 flags.go:64] FLAG: --runonce="false" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003134 4312 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003137 4312 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003140 4312 flags.go:64] FLAG: --seccomp-default="false" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003143 4312 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003147 4312 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003150 4312 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003153 4312 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003156 4312 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003159 4312 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003162 4312 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003165 4312 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003168 4312 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003171 4312 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003174 4312 flags.go:64] FLAG: --system-cgroups="" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003177 4312 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003183 4312 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 11:56:23.012428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003186 4312 flags.go:64] FLAG: --tls-cert-file="" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003188 4312 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003192 4312 flags.go:64] FLAG: --tls-min-version="" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003195 4312 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003197 4312 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003200 4312 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003203 4312 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003206 4312 flags.go:64] FLAG: --v="2" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003210 4312 flags.go:64] FLAG: --version="false" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003214 4312 flags.go:64] FLAG: --vmodule="" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003220 4312 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003223 4312 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003348 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003352 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003356 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003359 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003363 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003366 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003369 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003372 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003374 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003377 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:56:23.013086 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003379 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003382 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003385 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003388 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003390 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003393 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003395 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003398 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003401 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003403 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003406 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003409 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003411 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003413 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003416 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003419 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003421 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003424 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003426 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003429 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:56:23.013820 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003433 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003435 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003438 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003441 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003443 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003446 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003450 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003453 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003456 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003458 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003461 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003463 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003466 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003469 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003471 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003474 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003476 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003479 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003482 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:56:23.014457 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003484 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003487 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003489 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003505 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003508 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003511 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003514 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003516 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003523 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003525 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003528 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003531 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003533 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003537 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003542 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003547 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003552 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003557 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003561 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003565 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:56:23.015018 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003568 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003570 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003574 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003577 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003579 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003582 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003585 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003587 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003590 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003593 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003595 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003598 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003602 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003607 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003610 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003612 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003615 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003618 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003620 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:56:23.015522 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003623 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003625 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003628 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003631 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003633 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003636 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003639 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003642 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003645 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003647 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003650 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003652 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003657 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003659 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003662 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003664 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003667 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:56:23.016100 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003670 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.003696 4312 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003729 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003733 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003736 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003739 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003742 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003744 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003747 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003749 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003752 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003754 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003757 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003760 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003762 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003765 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:56:23.016678 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003767 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003770 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003772 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003774 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003777 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003781 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003783 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003786 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003789 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003792 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003795 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003797 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003801 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003804 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003806 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003809 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003812 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003814 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003817 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003820 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:56:23.017090 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003822 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003825 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003827 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003830 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003833 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003835 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003838 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003840 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003843 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003845 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003848 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003850 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003853 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003856 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003858 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003861 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003863 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003867 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003869 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003872 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:56:23.017613 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003874 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003877 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003879 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003882 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003885 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003888 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003891 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003893 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003896 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003899 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003902 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003904 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003907 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003909 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003912 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003914 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003917 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003919 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003922 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:56:23.018145 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003924 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003927 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003929 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003932 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003935 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003937 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003940 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003942 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003945 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003948 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003955 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003958 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003962 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003964 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003968 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003972 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003975 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003979 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003981 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003984 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:56:23.018647 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003987 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003990 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003992 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003996 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.003998 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.004001 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.004003 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.004006 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.004009 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.004012 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.004015 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.004019 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.004024 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.009025 4312 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.009039 4312 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.009054 4312 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.009058 4312 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009115 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009121 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:56:23.019134 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009124 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009127 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009130 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009133 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009136 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009138 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009142 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009144 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009147 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009150 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009153 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009156 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009158 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009161 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009163 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009166 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009169 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009172 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009174 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009177 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:56:23.019639 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009179 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009182 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009185 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009188 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009190 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009193 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009196 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009199 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009201 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009204 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009206 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009209 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009212 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009214 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009217 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009219 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009222 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009225 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009228 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:56:23.020116 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009230 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009233 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009236 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009238 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009241 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009244 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009247 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009250 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009252 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009255 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009257 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009260 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009262 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009265 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009267 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009270 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009272 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009275 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009277 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:56:23.020589 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009280 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009283 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009285 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009288 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009290 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009293 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009296 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009298 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009301 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009303 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009306 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009309 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009312 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009315 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009318 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009320 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009323 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009327 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009331 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:56:23.021151 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009335 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009339 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009342 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009344 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009347 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009350 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009352 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009355 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009358 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009360 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009363 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009365 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009368 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009370 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009373 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009375 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009378 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009381 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009390 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009394 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:56:23.021926 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009396 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009399 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009402 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009404 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009407 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009409 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009413 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.009441 4312 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009592 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009599 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009602 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009605 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009608 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009611 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009614 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:56:23.022863 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009616 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009619 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009622 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009624 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009627 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009630 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009633 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009635 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009638 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009641 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009643 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009647 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009649 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009652 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009655 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009657 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009659 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009662 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009665 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009668 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:56:23.023245 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009670 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009672 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009675 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009678 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009681 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009683 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009686 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009688 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009691 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009693 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009696 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009699 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009702 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009705 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009709 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009713 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009716 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009719 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009721 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:56:23.023749 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009724 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009726 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009729 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009732 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009735 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009737 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009740 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009743 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009746 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009748 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009751 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009753 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009756 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009759 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009762 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009766 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009769 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009772 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009775 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009777 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:56:23.024221 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009780 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009782 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009785 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009788 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009790 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009793 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009796 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009799 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009801 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009803 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009806 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009808 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009811 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009813 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009816 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009819 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009822 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009824 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009827 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009830 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:56:23.024711 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009832 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009835 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009837 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009840 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009843 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009845 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009848 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009850 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009853 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009855 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009858 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009861 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009864 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009867 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009869 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009872 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009875 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009878 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009881 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:56:23.025248 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009883 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.009906 4312 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009939 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009943 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009946 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009949 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009952 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009955 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009958 4312 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009960 4312 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009963 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009965 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009968 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009971 4312 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009974 4312 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009978 4312 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 11:56:23.026132 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009981 4312 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009984 4312 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009987 4312 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009989 4312 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009992 4312 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009994 4312 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009997 4312 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.009999 4312 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010002 4312 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010004 4312 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010007 4312 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010009 4312 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010012 4312 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010014 4312 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010017 4312 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010019 4312 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010022 4312 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010025 4312 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010028 4312 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010033 4312 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 11:56:23.026804 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010035 4312 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010038 4312 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010041 4312 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010043 4312 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010046 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010048 4312 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010051 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010053 4312 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010056 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010059 4312 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010061 4312 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010064 4312 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010066 4312 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010071 4312 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010074 4312 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010077 4312 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010079 4312 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010082 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010085 4312 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 11:56:23.027733 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010087 4312 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010090 4312 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010093 4312 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010095 4312 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010098 4312 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010100 4312 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010103 4312 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010105 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010108 4312 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010111 4312 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010114 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010117 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010121 4312 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010124 4312 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010129 4312 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010132 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010136 4312 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010138 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010141 4312 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 11:56:23.028190 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010144 4312 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010146 4312 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010149 4312 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010152 4312 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010154 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010157 4312 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010159 4312 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010164 4312 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010167 4312 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010170 4312 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010173 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010175 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010178 4312 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010181 4312 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010183 4312 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010186 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010189 4312 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010191 4312 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010194 4312 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010196 4312 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 11:56:23.028675 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010199 4312 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010201 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010204 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010207 4312 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010210 4312 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010214 4312 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010219 4312 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010223 4312 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010226 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010229 4312 feature_gate.go:386] unrecognized feature gate: Example Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010232 4312 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010235 4312 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010237 4312 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:23.010240 4312 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.010292 4312 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.011262 4312 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.011320 4312 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.011510 4312 server.go:1014] "Starting client certificate rotation" Nov 02 11:56:23.029159 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.011611 4312 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 11:56:23.029620 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.011676 4312 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 11:50:43 +0000 UTC" deadline="2028-07-05 10:21:32.407789181 +0000 UTC" Nov 02 11:56:23.029620 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.011725 4312 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14662h25m9.396067757s" Nov 02 11:56:23.029620 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.011964 4312 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 11:56:23.029620 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.012996 4312 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 11:56:23.029620 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.014329 4312 log.go:25] "Validated CRI v1 runtime API" Nov 02 11:56:23.029620 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.015739 4312 log.go:25] "Validated CRI v1 image API" Nov 02 11:56:23.029620 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.016469 4312 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 11:56:23.029620 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.017767 4312 fs.go:135] Filesystem UUIDs: map[15f751de-ad9e-45dc-b2b6-dd4cf415daba:/dev/nvme0n1p3 3923697a-5ca7-4ed4-be00-97f60c35a80f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.017787 4312 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18f2d976231ccb21e879c9163ef4ccb85dee26fb1a603689982c91d2adffeff9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18f2d976231ccb21e879c9163ef4ccb85dee26fb1a603689982c91d2adffeff9/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e2c0b22d038233964e850e642f73c6fd514e9f4ffcb67ad22f0eae85e4c4d7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e2c0b22d038233964e850e642f73c6fd514e9f4ffcb67ad22f0eae85e4c4d7d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3175751de3169d589d580a270de77ebd96dc2d79e201f8781d78cab718800358/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3175751de3169d589d580a270de77ebd96dc2d79e201f8781d78cab718800358/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b44493826af1f4208087805fe2e5f3256ebbf4dda2a7b23f0949a1c8469c4ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b44493826af1f4208087805fe2e5f3256ebbf4dda2a7b23f0949a1c8469c4ad/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e9fecd00c843db3b4d4eb945c394104d4467f12b2259104065c7aac804e6f35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e9fecd00c843db3b4d4eb945c394104d4467f12b2259104065c7aac804e6f35/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95793f9e8069e19773f8c6fe37bee884bc0428fe789ee04e369af63f13649b6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95793f9e8069e19773f8c6fe37bee884bc0428fe789ee04e369af63f13649b6f/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c64235b0453ba3de6be7227ff41aa509ff11973ffa307f1a461fbc048d322e11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c64235b0453ba3de6be7227ff41aa509ff11973ffa307f1a461fbc048d322e11/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c967aa12c33d48ee034395bde5e24b790035563ce2d2101c5b2798bbcc8f94ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c967aa12c33d48ee034395bde5e24b790035563ce2d2101c5b2798bbcc8f94ad/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0edb3240de6ce97514a61f70b102c54d747a84e061ec920a30bf406748872fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0edb3240de6ce97514a61f70b102c54d747a84e061ec920a30bf406748872fe/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df67a2733206734031c5297f313c4094f46c387533c5849890aa9501dd085c4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df67a2733206734031c5297f313c4094f46c387533c5849890aa9501dd085c4d/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e04937d447fc02f733c0644281139b1bad1e426678ab6fb0bcfc668fc079bcc4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e04937d447fc02f733c0644281139b1bad1e426678ab6fb0bcfc668fc079bcc4/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee6575e8525fa55b0a75ff3404575760b419430d526880e0722fe793bfd998ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee6575e8525fa55b0a75ff3404575760b419430d526880e0722fe793bfd998ac/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05fb7da7-c7b3-412e-a14e-45d7c10d4216/volumes/kubernetes.io~projected/kube-api-access-8686b:{mountpoint:/var/lib/kubelet/pods/05fb7da7-c7b3-412e-a14e-45d7c10d4216/volumes/kubernetes.io~projected/kube-api-access-8686b major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bcac010-ce3f-4fac-a5a0-5833466e4bd8/volumes/kubernetes.io~projected/kube-api-access-8kc7z:{mountpoint:/var/lib/kubelet/pods/1bcac010-ce3f-4fac-a5a0-5833466e4bd8/volumes/kubernetes.io~projected/kube-api-access-8kc7z major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c5d3609-6aef-4c1f-a876-6cb15457cc9a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1c5d3609-6aef-4c1f-a876-6cb15457cc9a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c5d3609-6aef-4c1f-a876-6cb15457cc9a/volumes/kubernetes.io~projected/kube-api-access-jbbqq:{mountpoint:/var/lib/kubelet/pods/1c5d3609-6aef-4c1f-a876-6cb15457cc9a/volumes/kubernetes.io~projected/kube-api-access-jbbqq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c5d3609-6aef-4c1f-a876-6cb15457cc9a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1c5d3609-6aef-4c1f-a876-6cb15457cc9a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/291cad74-3a77-43f6-abb4-d28aaa5eb20c/volumes/kubernetes.io~projected/kube-api-access-c5wc6:{mountpoint:/var/lib/kubelet/pods/291cad74-3a77-43f6-abb4-d28aaa5eb20c/volumes/kubernetes.io~projected/kube-api-access-c5wc6 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44d668e0-e534-4e0b-8d32-31ead2f678a2/volumes/kubernetes.io~projected/kube-api-access-hhlnz:{mountpoint:/var/lib/kubelet/pods/44d668e0-e534-4e0b-8d32-31ead2f678a2/volumes/kubernetes.io~projected/kube-api-access-hhlnz major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b/volumes/kubernetes.io~projected/kube-api-access-xqh2j:{mountpoint:/var/lib/kubelet/pods/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b/volumes/kubernetes.io~projected/kube-api-access-xqh2j major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bebca75-b1fd-4e59-9a4b-d6589df81d9b/volumes/kubernetes.io~projected/kube-api-access-6bctf:{mountpoint:/var/lib/kubelet/pods/5bebca75-b1fd-4e59-9a4b-d6589df81d9b/volumes/kubernetes.io~projected/kube-api-access-6bctf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6533ee11-4dc9-4c0f-b035-1200f74d869b/volumes/kubernetes.io~projected/kube-api-access-hhq46:{mountpoint:/var/lib/kubelet/pods/6533ee11-4dc9-4c0f-b035-1200f74d869b/volumes/kubernetes.io~projected/kube-api-access-hhq46 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e55d04a-613c-4e3a-bcc5-32fea4e50693/volumes/kubernetes.io~projected/kube-api-access-djtqw:{mountpoint:/var/lib/kubelet/pods/7e55d04a-613c-4e3a-bcc5-32fea4e50693/volumes/kubernetes.io~projected/kube-api-access-djtqw major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f547b23-0439-4cc3-9127-387407a34b31/volumes/kubernetes.io~projected/kube-api-access-75mwz:{mountpoint:/var/lib/kubelet/pods/7f547b23-0439-4cc3-9127-387407a34b31/volumes/kubernetes.io~projected/kube-api-access-75mwz major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/837a6c92-ea3e-490a-810e-05e2fc91a89a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/837a6c92-ea3e-490a-810e-05e2fc91a89a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcda3ab4-1f37-49db-8eb9-2b83fe89d484/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/fcda3ab4-1f37-49db-8eb9-2b83fe89d484/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/2113ecfb8a3ba9f6cb6e6d1f3a48483d453f7cbac180271af43ec213c52a45e1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6e5873006eff29f3e4a9ce832c0d06d0ede8b585ecc2e61da0c992efe025c37b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/2e3b703a8f002a975509bc22d842ca8193a0eb492f47a92a9c446441d08972dd/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/a9477fbeeff191a7efeb36ac3fb3242c579aa0972afb590ebed25e32d07f57a0/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/28d2860d6a4c90c2009d465c5c36812b0255404ffdbdc279f5419ede89d06df3/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1221bd43ed11dab3a8423758d5613894fb7843697bde48b951ecbf978fb24007/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8d491b75cb9c1b5b9b0073371a9528a938504dfd8a043b07e3eee7701ecd57af/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/8ebc8e56b9069ca48fc146105531b26a052ab910375bb51b69980c643c5ff053/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/cb4b7edf7465fccc621ef5c825e11cf695ab83886f75a45e0ab94c49936a3c96/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/b1065df208b47a8f853be7f2539f4e09237e2f12f96018d840f802213ef98aa4/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/7b2f368810eaaf2b394a183bc005407cb4c1b8bdbed3dd8a699d28e1e25be244/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/278f152d91716f5665608a5d1104fde4240db7f05d78c66e69a5468b91a68006/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1d8e0c927e8088511ec120ffc315a406a85aa0328b22b0302cd821a0c3a28434/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ff4d46072864dc75ea8c8b37d4a8109c3ccc1022120a75d6a3b58379767550e1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/09a6eb7f765c4681a9a2cfa46d084ff85f00ccb3d1dd87707b6e697a0515a826/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/04555f7f08cf7f9fafae94cc404d4618f712774248d1d73dcbbc79992cc91561/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2c242500abededbf0deacc1cd14c714fb260968a3cc628b16cd6f39338ef5b16/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/59a116c4a1bb2d7f32e5b8df17b45f94cecf45bcc04a3ddf017808a116ee69ce/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0a3d28f9f552697c518b5c74df307d16cdf11f5cc6b4544c67683d39018fc51f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/6ff4dcba4230c002c53b1da67f114e819c9e95ec14fee56ce22701b685fdd951/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7e89e1185ec0d271d2226bcacab2324b45af87f459c7bd2ef31cddbb6488e310/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/d9b98fd0da6ff1d83d9479cd3e517dc6079e56538cbdfae8d7bd036b0eba97a8/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/3c349c28b1538f065b17a3ecd17113c630226c4633102f70eb7fdaac6a108e41/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/9dfa169b52cc3cdea4c23466c9a2c5b38f484b4058fbd73f1c80eb1ece077b7e/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/8045f434e38f0f86e876ba297dadc6bec8c788a546ba4d4dab5f1f4e50725294/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/841e159267f80619ede7616b7ff8803d9849679843e62e0014d231debb4cf3e5/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b0a118d7ef3b7878a822206ba33f726e61c88f7da6fba4116c567c73bf284c63/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/80e96b3a60b1fbde277b51f5fb3c6923a85b7c3c07286c79344c5a9c6cee3240/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a787d79812f3c33a4981a3593023983594d7e3908e4ed6acbe1d250ae93c5ae3/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/602f7db942a28d61f46bd7cd7309937de05199c6e810a2ed69fa87598f2c343f/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/557952a6e834654dd6d53032e49e5d17330d75299691b5d049277d00fc39a124/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/279d85263c70620a4fbf87cf6d7a783e9af21e286f33dda140aba56263f7727e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/7f6c2b92d13fde9a83d2146198b8f184b2d41f3fb585f9364241b7ff7f3b30bc/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/e872c28cfc5094aca0d3d8b721c3fcd9756db6995c071dbff4c6d83f55e0a4a0/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9bf017fdc6238f5244a78a6d080014574d840acfbdef2aea883be46faa4dea50/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.024815 4312 manager.go:217] Machine: {Timestamp:2026-11-02 11:56:23.024352745 +0000 UTC m=+0.054588986 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101858 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23d1f5cd06312ed0d81395841974a9 SystemUUID:ec23d1f5-cd06-312e-d0d8-1395841974a9 BootID:27f4c9f6-d5a9-473b-94f4-88ab9c22ab6b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c967aa12c33d48ee034395bde5e24b790035563ce2d2101c5b2798bbcc8f94ad/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b44493826af1f4208087805fe2e5f3256ebbf4dda2a7b23f0949a1c8469c4ad/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcda3ab4-1f37-49db-8eb9-2b83fe89d484/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95793f9e8069e19773f8c6fe37bee884bc0428fe789ee04e369af63f13649b6f/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c5d3609-6aef-4c1f-a876-6cb15457cc9a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05fb7da7-c7b3-412e-a14e-45d7c10d4216/volumes/kubernetes.io~projected/kube-api-access-8686b DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df67a2733206734031c5297f313c4094f46c387533c5849890aa9501dd085c4d/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee6575e8525fa55b0a75ff3404575760b419430d526880e0722fe793bfd998ac/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e55d04a-613c-4e3a-bcc5-32fea4e50693/volumes/kubernetes.io~projected/kube-api-access-djtqw DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e9fecd00c843db3b4d4eb945c394104d4467f12b2259104065c7aac804e6f35/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18f2d976231ccb21e879c9163ef4ccb85dee26fb1a603689982c91d2adffeff9/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3175751de3169d589d580a270de77ebd96dc2d79e201f8781d78cab718800358/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1bcac010-ce3f-4fac-a5a0-5833466e4bd8/volumes/kubernetes.io~projected/kube-api-access-8kc7z DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/291cad74-3a77-43f6-abb4-d28aaa5eb20c/volumes/kubernetes.io~projected/kube-api-access-c5wc6 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e04937d447fc02f733c0644281139b1bad1e426678ab6fb0bcfc668fc079bcc4/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f547b23-0439-4cc3-9127-387407a34b31/volumes/kubernetes.io~projected/kube-api-access-75mwz DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0edb3240de6ce97514a61f70b102c54d747a84e061ec920a30bf406748872fe/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e2c0b22d038233964e850e642f73c6fd514e9f4ffcb67ad22f0eae85e4c4d7d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6533ee11-4dc9-4c0f-b035-1200f74d869b/volumes/kubernetes.io~projected/kube-api-access-hhq46 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/5bebca75-b1fd-4e59-9a4b-d6589df81d9b/volumes/kubernetes.io~projected/kube-api-access-6bctf DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c5d3609-6aef-4c1f-a876-6cb15457cc9a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/1c5d3609-6aef-4c1f-a876-6cb15457cc9a/volumes/kubernetes.io~projected/kube-api-access-jbbqq DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b/volumes/kubernetes.io~projected/kube-api-access-xqh2j DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/837a6c92-ea3e-490a-810e-05e2fc91a89a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/44d668e0-e534-4e0b-8d32-31ead2f678a2/volumes/kubernetes.io~projected/kube-api-access-hhlnz DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c64235b0453ba3de6be7227ff41aa509ff11973ffa307f1a461fbc048d322e11/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:18f2d976231ccb2 MacAddress:d6:25:71:cb:20:e5 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:01:41:11:50:a3 Speed:0 Mtu:9001} {Name:br-int MacAddress:ae:bd:e1:89:67:12 Speed:0 Mtu:8901} {Name:c64235b0453ba3d MacAddress:a2:e9:6f:32:b6:53 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:01:41:11:50:a3 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:be:ae:7f:df:1d:67 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:e6:44:7b:77:86:a5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.025030 4312 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.025068 4312 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.025476 4312 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.025524 4312 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-170.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.025715 4312 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.025727 4312 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.025741 4312 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.025769 4312 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.025879 4312 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.026088 4312 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.026231 4312 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.026282 4312 kubelet.go:493] "Attempting to sync node with API server" Nov 02 11:56:23.029830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.026300 4312 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.026314 4312 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.026326 4312 kubelet.go:405] "Adding apiserver pod source" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.026337 4312 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.027151 4312 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.027164 4312 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.027681 4312 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029457 4312 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029466 4312 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029570 4312 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029737 4312 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029763 4312 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029861 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029872 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029878 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029883 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029889 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029895 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029901 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029907 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029917 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029923 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029935 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029943 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029984 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 11:56:23.030378 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.029989 4312 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 11:56:23.033027 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.033014 4312 server.go:1263] "Started kubelet" Nov 02 11:56:23.033516 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.033093 4312 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 11:56:23.033516 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.033114 4312 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 11:56:23.033516 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.033200 4312 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 11:56:23.033544 ip-10-0-141-170 systemd[1]: Started Kubernetes Kubelet. Nov 02 11:56:23.033876 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.033820 4312 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 11:56:23.034393 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.034378 4312 server.go:317] "Adding debug handlers to kubelet server" Nov 02 11:56:23.038403 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.038384 4312 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 11:56:23.038565 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.038547 4312 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 11:56:23.038627 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:23.038609 4312 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:56:23.038698 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.038620 4312 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 11:50:44 +0000 UTC" deadline="2028-04-19 08:00:35.544664102 +0000 UTC" Nov 02 11:56:23.038698 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.038655 4312 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12812h4m12.506014466s" Nov 02 11:56:23.038698 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.038657 4312 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 11:56:23.038698 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.038644 4312 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 11:56:23.038698 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.038695 4312 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 11:56:23.039833 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039806 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c5d3609-6aef-4c1f-a876-6cb15457cc9a" volumeName="kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-config" seLinuxMountContext="" Nov 02 11:56:23.039833 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039833 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="291cad74-3a77-43f6-abb4-d28aaa5eb20c" volumeName="kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039843 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bebca75-b1fd-4e59-9a4b-d6589df81d9b" volumeName="kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cni-binary-copy" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039851 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bebca75-b1fd-4e59-9a4b-d6589df81d9b" volumeName="kubernetes.io/projected/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-kube-api-access-6bctf" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039859 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6533ee11-4dc9-4c0f-b035-1200f74d869b" volumeName="kubernetes.io/projected/6533ee11-4dc9-4c0f-b035-1200f74d869b-kube-api-access-hhq46" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039866 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcda3ab4-1f37-49db-8eb9-2b83fe89d484" volumeName="kubernetes.io/configmap/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-konnectivity-ca" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039874 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59ecdb0c-5e72-4d63-b737-a9c222dc8d2b" volumeName="kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-tmp" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039884 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05fb7da7-c7b3-412e-a14e-45d7c10d4216" volumeName="kubernetes.io/projected/05fb7da7-c7b3-412e-a14e-45d7c10d4216-kube-api-access-8686b" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039898 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bcac010-ce3f-4fac-a5a0-5833466e4bd8" volumeName="kubernetes.io/projected/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kube-api-access-8kc7z" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039910 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c5d3609-6aef-4c1f-a876-6cb15457cc9a" volumeName="kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-script-lib" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039917 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c5d3609-6aef-4c1f-a876-6cb15457cc9a" volumeName="kubernetes.io/projected/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-kube-api-access-jbbqq" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039926 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c5d3609-6aef-4c1f-a876-6cb15457cc9a" volumeName="kubernetes.io/secret/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039933 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59ecdb0c-5e72-4d63-b737-a9c222dc8d2b" volumeName="kubernetes.io/projected/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-kube-api-access-xqh2j" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039939 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bebca75-b1fd-4e59-9a4b-d6589df81d9b" volumeName="kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-daemon-config" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039949 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44d668e0-e534-4e0b-8d32-31ead2f678a2" volumeName="kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039955 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="59ecdb0c-5e72-4d63-b737-a9c222dc8d2b" volumeName="kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-tuned" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039962 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6533ee11-4dc9-4c0f-b035-1200f74d869b" volumeName="kubernetes.io/configmap/6533ee11-4dc9-4c0f-b035-1200f74d869b-iptables-alerter-script" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039969 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e55d04a-613c-4e3a-bcc5-32fea4e50693" volumeName="kubernetes.io/configmap/7e55d04a-613c-4e3a-bcc5-32fea4e50693-serviceca" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039976 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f547b23-0439-4cc3-9127-387407a34b31" volumeName="kubernetes.io/empty-dir/7f547b23-0439-4cc3-9127-387407a34b31-tmp-dir" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039982 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f547b23-0439-4cc3-9127-387407a34b31" volumeName="kubernetes.io/projected/7f547b23-0439-4cc3-9127-387407a34b31-kube-api-access-75mwz" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.039993 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcda3ab4-1f37-49db-8eb9-2b83fe89d484" volumeName="kubernetes.io/secret/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-agent-certs" seLinuxMountContext="" Nov 02 11:56:23.040006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.040004 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="291cad74-3a77-43f6-abb4-d28aaa5eb20c" volumeName="kubernetes.io/projected/291cad74-3a77-43f6-abb4-d28aaa5eb20c-kube-api-access-c5wc6" seLinuxMountContext="" Nov 02 11:56:23.040805 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.040017 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c5d3609-6aef-4c1f-a876-6cb15457cc9a" volumeName="kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-env-overrides" seLinuxMountContext="" Nov 02 11:56:23.040805 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.040029 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="291cad74-3a77-43f6-abb4-d28aaa5eb20c" volumeName="kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-binary-copy" seLinuxMountContext="" Nov 02 11:56:23.040805 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.040041 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="291cad74-3a77-43f6-abb4-d28aaa5eb20c" volumeName="kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 11:56:23.040805 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.040053 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e55d04a-613c-4e3a-bcc5-32fea4e50693" volumeName="kubernetes.io/projected/7e55d04a-613c-4e3a-bcc5-32fea4e50693-kube-api-access-djtqw" seLinuxMountContext="" Nov 02 11:56:23.040805 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.040066 4312 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="837a6c92-ea3e-490a-810e-05e2fc91a89a" volumeName="kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret" seLinuxMountContext="" Nov 02 11:56:23.040805 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.040078 4312 reconstruct.go:97] "Volume reconstruction finished" Nov 02 11:56:23.040805 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.040086 4312 reconciler.go:29] "Reconciler: start to sync state" Nov 02 11:56:23.041135 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.041111 4312 factory.go:55] Registering systemd factory Nov 02 11:56:23.041135 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.041127 4312 factory.go:223] Registration of the systemd container factory successfully Nov 02 11:56:23.041324 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.041309 4312 factory.go:153] Registering CRI-O factory Nov 02 11:56:23.041376 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.041323 4312 factory.go:223] Registration of the crio container factory successfully Nov 02 11:56:23.041433 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.041380 4312 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 11:56:23.041433 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.041408 4312 factory.go:103] Registering Raw factory Nov 02 11:56:23.041433 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.041422 4312 manager.go:1199] Started watching for new ooms in manager Nov 02 11:56:23.041796 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.041783 4312 manager.go:319] Starting recovery of all containers Nov 02 11:56:23.066795 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.066766 4312 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 11:56:23.068079 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.068056 4312 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 11:56:23.068079 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.068081 4312 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 11:56:23.068208 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.068098 4312 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 11:56:23.068208 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:23.068129 4312 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 11:56:23.082547 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.082488 4312 generic.go:356] "Generic (PLEG): container finished" podID="14b5e76aab2a7b6af2007243418d856f" containerID="50657f8c433699736a7ec72876c559843cb98968d8f8e868fc90a9d9415d51e0" exitCode=0 Nov 02 11:56:23.091744 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.091719 4312 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="e231929b133c224219969be6f8d8e0a64605eabcd252a48113eeb8c174c091df" exitCode=0 Nov 02 11:56:23.091744 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.091741 4312 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="ffb22e091dbc5b383057a78c1a867dd96cc382a22ef3fe7656fd6326a9f85ff8" exitCode=0 Nov 02 11:56:23.091907 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.091754 4312 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="a6b506fb2ffc5103610d4e87908d89491eee01c95b9a0797eee060f8a635d780" exitCode=0 Nov 02 11:56:23.091907 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.091761 4312 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="6627f0fa0008579dbbcf9caf6198766919d5cbec806bdc29a1d74af7e0dff6b1" exitCode=0 Nov 02 11:56:23.091907 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.091772 4312 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="38fc2a3de5db76b3cef5f263bc10bb2d01f0e24423b2ba1e1282f3c05c580f41" exitCode=0 Nov 02 11:56:23.091907 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.091780 4312 generic.go:356] "Generic (PLEG): container finished" podID="291cad74-3a77-43f6-abb4-d28aaa5eb20c" containerID="0186e52975365ac8864dcad8376abc551977143341a77a14897c7f0944467e7f" exitCode=0 Nov 02 11:56:23.097435 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.097418 4312 manager.go:324] Recovery completed Nov 02 11:56:23.109749 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.109731 4312 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:56:23.110728 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.110714 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:56:23.110793 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.110739 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:56:23.110793 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.110750 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:56:23.111827 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.111813 4312 cpu_manager.go:226] "Starting" policy="none" Nov 02 11:56:23.111827 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.111825 4312 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 11:56:23.111907 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.111835 4312 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:56:23.111907 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.111840 4312 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 11:56:23.112048 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112033 4312 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 11:56:23.112079 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112049 4312 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 11:56:23.112079 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112064 4312 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 11:56:23.112079 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112069 4312 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 11:56:23.112079 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112074 4312 policy_none.go:50] "Start" Nov 02 11:56:23.112183 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112083 4312 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 11:56:23.112183 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112094 4312 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 11:56:23.112236 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112212 4312 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 11:56:23.112236 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112219 4312 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 11:56:23.112236 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112224 4312 policy_none.go:44] "Start" Nov 02 11:56:23.112622 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112608 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:56:23.112726 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112633 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:56:23.112726 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.112642 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:56:23.116060 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.116037 4312 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 11:56:23.119007 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.118996 4312 manager.go:341] "Starting Device Plugin manager" Nov 02 11:56:23.119045 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:23.119019 4312 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 11:56:23.119045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.119028 4312 server.go:87] "Starting device plugin registration server" Nov 02 11:56:23.119317 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.119305 4312 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 11:56:23.119353 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.119319 4312 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 11:56:23.119423 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.119405 4312 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 11:56:23.119550 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.119517 4312 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 11:56:23.119641 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.119554 4312 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 11:56:23.119641 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.119562 4312 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 11:56:23.119805 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.119595 4312 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T11:56:23.11954409Z","UUID":"311a7fb6-83ad-4a3b-920d-9f04f09b0729","Handler":null,"Name":"","Endpoint":""} Nov 02 11:56:23.120629 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:23.120600 4312 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 11:56:23.120706 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:23.120641 4312 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" Nov 02 11:56:23.120829 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.120815 4312 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 11:56:23.120895 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.120843 4312 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 11:56:23.168863 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.168383 4312 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal"] Nov 02 11:56:23.168956 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.168945 4312 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:56:23.169719 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.169687 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:56:23.169719 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.169720 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:56:23.169843 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.169734 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:56:23.169983 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.169970 4312 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:56:23.170112 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.170098 4312 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:56:23.170721 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.170703 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:56:23.170721 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.170715 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:56:23.170840 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.170736 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:56:23.170840 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.170748 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:56:23.170840 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.170736 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:56:23.170840 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.170779 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:56:23.170948 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:23.170872 4312 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" node="ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.170995 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.170969 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" event={"ID":"14b5e76aab2a7b6af2007243418d856f","Type":"ContainerStarted","Data":"86c19a00a41adfe03b0d6a13adbc6aa920acb1a05b636c69557525f035a6ea8b"} Nov 02 11:56:23.171025 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.171002 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" event={"ID":"14b5e76aab2a7b6af2007243418d856f","Type":"ContainerDied","Data":"50657f8c433699736a7ec72876c559843cb98968d8f8e868fc90a9d9415d51e0"} Nov 02 11:56:23.171025 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.171019 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" event={"ID":"14b5e76aab2a7b6af2007243418d856f","Type":"ContainerStarted","Data":"3175751de3169d589d580a270de77ebd96dc2d79e201f8781d78cab718800358"} Nov 02 11:56:23.171081 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.171036 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" event={"ID":"1ebeef05fd06ac44ab33873433450b74","Type":"ContainerStarted","Data":"514cbe11b9da4fe6a7927abdfa89423d2ad8ab9b88833f7fa1047c56af4e4e42"} Nov 02 11:56:23.171081 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.171047 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" event={"ID":"1ebeef05fd06ac44ab33873433450b74","Type":"ContainerStarted","Data":"1e2c0b22d038233964e850e642f73c6fd514e9f4ffcb67ad22f0eae85e4c4d7d"} Nov 02 11:56:23.171081 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.171055 4312 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:56:23.171734 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.171720 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:56:23.171779 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.171742 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:56:23.171779 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.171752 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:56:23.171867 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:23.171856 4312 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-170.us-west-1.compute.internal\" not found" node="ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.219710 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.219690 4312 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 11:56:23.220391 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.220373 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 11:56:23.220476 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.220402 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 11:56:23.220476 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.220413 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 11:56:23.220476 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.220427 4312 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.237319 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.237296 4312 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 11:56:23.237319 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.237310 4312 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 11:56:23.237472 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.237397 4312 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 11:56:23.285500 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.285483 4312 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 11:56:23.287140 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.287119 4312 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 11:56:23.338064 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.338043 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.338136 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.338067 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.338136 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.338084 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ebeef05fd06ac44ab33873433450b74-config\") pod \"kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"1ebeef05fd06ac44ab33873433450b74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.339018 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.338997 4312 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.339293 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.339272 4312 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.339359 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.339347 4312 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.341102 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.341085 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeNotReady" Nov 02 11:56:23.341489 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.341144 4312 setters.go:546] "Node became not ready" node="ip-10-0-141-170.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T11:56:23Z","lastTransitionTime":"2026-11-02T11:56:23Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 11:56:23.387617 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.387592 4312 nodeinfomanager.go:471] existing CSINode "ip-10-0-141-170.us-west-1.compute.internal" is owned by different node (oldNodeID="96bbd084-f13b-423e-b669-3c4f5cc5eb86", newNodeID=""), cleaning up... Nov 02 11:56:23.399900 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.399883 4312 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 11:56:23.399949 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:23.399915 4312 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.399949 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.399925 4312 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.438481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.438431 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.438481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.438459 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.438481 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.438477 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ebeef05fd06ac44ab33873433450b74-config\") pod \"kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"1ebeef05fd06ac44ab33873433450b74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.438659 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.438525 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1ebeef05fd06ac44ab33873433450b74-config\") pod \"kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"1ebeef05fd06ac44ab33873433450b74\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.438659 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.438569 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.438659 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.438567 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/14b5e76aab2a7b6af2007243418d856f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" (UID: \"14b5e76aab2a7b6af2007243418d856f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.457847 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.457825 4312 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 11:56:23.457925 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:23.457860 4312 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:23.609729 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.609710 4312 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-170.us-west-1.compute.internal" event="NodeReady" Nov 02 11:56:23.609818 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:23.609771 4312 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 11:56:24.026728 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.026709 4312 apiserver.go:52] "Watching apiserver" Nov 02 11:56:24.100539 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.100517 4312 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:24.100655 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.100640 4312 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:24.128812 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.128791 4312 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 11:56:24.129113 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.129097 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-2wrsc","kube-system/konnectivity-agent-5pnd2","kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal","openshift-dns/node-resolver-nnpfg","openshift-multus/multus-w4h5q","openshift-multus/network-metrics-daemon-p2frl","openshift-network-operator/iptables-alerter-4ncdp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42","openshift-image-registry/node-ca-np85r","openshift-network-diagnostics/network-check-target-s2fqn","kube-system/global-pull-secret-syncer-d5dpl","openshift-cluster-node-tuning-operator/tuned-gnt4x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-mmzdg"] Nov 02 11:56:24.130751 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.130729 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:24.139670 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.139651 4312 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 11:56:24.143159 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143144 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhq46\" (UniqueName: \"kubernetes.io/projected/6533ee11-4dc9-4c0f-b035-1200f74d869b-kube-api-access-hhq46\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:56:24.143227 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143170 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-bin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.143227 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143188 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-kubelet\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.143227 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143202 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-node-log\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.143227 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143218 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbbqq\" (UniqueName: \"kubernetes.io/projected/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-kube-api-access-jbbqq\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.143343 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143255 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-kubelet-config\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:24.143386 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143373 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e55d04a-613c-4e3a-bcc5-32fea4e50693-host\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:56:24.143417 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143392 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-netd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.143417 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143407 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-dbus\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:24.143473 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143432 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5wc6\" (UniqueName: \"kubernetes.io/projected/291cad74-3a77-43f6-abb4-d28aaa5eb20c-kube-api-access-c5wc6\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.143512 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143466 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc7z\" (UniqueName: \"kubernetes.io/projected/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kube-api-access-8kc7z\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.143550 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143532 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6533ee11-4dc9-4c0f-b035-1200f74d869b-host-slash\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:56:24.143675 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143658 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f547b23-0439-4cc3-9127-387407a34b31-hosts-file\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:56:24.143705 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143687 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-daemon-config\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.143740 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143702 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-conf\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.143740 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143718 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.143740 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143735 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f547b23-0439-4cc3-9127-387407a34b31-tmp-dir\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:56:24.143824 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143752 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-conf-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.143824 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143787 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-log-socket\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.144011 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.143852 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f547b23-0439-4cc3-9127-387407a34b31-tmp-dir\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:56:24.144171 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144135 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-script-lib\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.144407 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144378 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-kubernetes\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.144461 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144444 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-tmp\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.144520 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144479 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.144587 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144571 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-ovn\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.144635 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144607 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-bin\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.144698 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144638 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-run\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.144698 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144678 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-socket-dir-parent\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.144790 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144737 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-konnectivity-ca\") pod \"konnectivity-agent-5pnd2\" (UID: \"fcda3ab4-1f37-49db-8eb9-2b83fe89d484\") " pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:56:24.144790 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144773 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhlnz\" (UniqueName: \"kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz\") pod \"network-check-target-s2fqn\" (UID: \"44d668e0-e534-4e0b-8d32-31ead2f678a2\") " pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:24.144873 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144813 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-netns\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.144873 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144846 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-host\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.144957 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144878 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8686b\" (UniqueName: \"kubernetes.io/projected/05fb7da7-c7b3-412e-a14e-45d7c10d4216-kube-api-access-8686b\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:24.144957 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144910 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtqw\" (UniqueName: \"kubernetes.io/projected/7e55d04a-613c-4e3a-bcc5-32fea4e50693-kube-api-access-djtqw\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:56:24.144957 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144937 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6533ee11-4dc9-4c0f-b035-1200f74d869b-iptables-alerter-script\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:56:24.145077 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.144969 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-system-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.145181 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145165 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-sys\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.145227 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145200 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-system-cni-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.145272 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145234 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75mwz\" (UniqueName: \"kubernetes.io/projected/7f547b23-0439-4cc3-9127-387407a34b31-kube-api-access-75mwz\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:56:24.145272 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145264 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-slash\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.145356 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145285 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-systemd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.145356 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145314 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-config\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.145356 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145343 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:24.145491 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145373 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.145491 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145413 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-etc-kubernetes\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.145491 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145437 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-systemd-units\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.145491 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145467 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.145680 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145518 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqh2j\" (UniqueName: \"kubernetes.io/projected/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-kube-api-access-xqh2j\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.145680 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145555 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:24.145680 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145586 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.145809 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145738 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-sys-fs\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.145809 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145767 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-hostroot\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.145809 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145799 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bctf\" (UniqueName: \"kubernetes.io/projected/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-kube-api-access-6bctf\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.145937 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145832 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-var-lib-kubelet\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.145937 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145857 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-etc-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.145937 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145886 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysconfig\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.145937 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145915 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-socket-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.146101 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145945 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cnibin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.146101 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.145976 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-agent-certs\") pod \"konnectivity-agent-5pnd2\" (UID: \"fcda3ab4-1f37-49db-8eb9-2b83fe89d484\") " pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:56:24.146101 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146004 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.146101 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146035 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-tuned\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.146101 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146066 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.146101 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146097 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-registration-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.146358 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146130 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.146358 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146155 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-k8s-cni-cncf-io\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.146358 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146184 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-netns\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.146358 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146214 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.146358 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146244 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-device-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.146358 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146275 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-multus-certs\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.146358 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146302 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-env-overrides\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.146358 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146333 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovn-node-metrics-cert\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.146741 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146361 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.146741 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146392 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-lib-modules\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.146741 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146418 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.146741 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146449 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-kubelet\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.146741 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146478 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-var-lib-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.146741 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146525 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-modprobe-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.146741 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146556 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cnibin\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.147013 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146862 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cni-binary-copy\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.147013 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146892 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-multus\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.147013 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.146968 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-tmp\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.147184 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.147163 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-systemd\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.147331 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.147311 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-tuned\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.147415 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.147399 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e55d04a-613c-4e3a-bcc5-32fea4e50693-serviceca\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:56:24.161044 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.161022 4312 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 11:56:24.161120 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:24.161061 4312 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:24.161544 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.161531 4312 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 11:56:24.161591 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:24.161566 4312 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-170.us-west-1.compute.internal" Nov 02 11:56:24.181984 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.181958 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mvmxd\"" Nov 02 11:56:24.182145 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.182127 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hc67r\"" Nov 02 11:56:24.182145 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.182131 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 11:56:24.182245 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.182201 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hx94g\"" Nov 02 11:56:24.182955 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.182939 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gfx2g\"" Nov 02 11:56:24.185129 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185108 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 11:56:24.185206 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185139 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 11:56:24.185206 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185144 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 11:56:24.185206 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185172 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 11:56:24.185206 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185174 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 11:56:24.185206 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185175 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 11:56:24.185426 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185230 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 11:56:24.185426 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185391 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 11:56:24.185426 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185397 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 11:56:24.185426 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185401 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 11:56:24.185627 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185446 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 11:56:24.185627 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185451 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 11:56:24.185627 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185484 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 11:56:24.185627 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185512 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 11:56:24.185627 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185517 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 11:56:24.185627 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185589 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 11:56:24.185627 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185622 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 11:56:24.185627 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185630 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 11:56:24.185852 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185637 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 11:56:24.185852 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185641 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 11:56:24.185852 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185674 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 11:56:24.185852 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185725 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 11:56:24.185852 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.185802 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 11:56:24.188198 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.188124 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-env-overrides\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.188275 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.188228 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-config\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.188337 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.188276 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.188337 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.188286 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cni-binary-copy\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.188337 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.188304 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cni-binary-copy\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.188337 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.188307 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/291cad74-3a77-43f6-abb4-d28aaa5eb20c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.188337 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.188288 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-konnectivity-ca\") pod \"konnectivity-agent-5pnd2\" (UID: \"fcda3ab4-1f37-49db-8eb9-2b83fe89d484\") " pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:56:24.188608 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.188591 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/837a6c92-ea3e-490a-810e-05e2fc91a89a-original-pull-secret\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:24.188690 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.188626 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e55d04a-613c-4e3a-bcc5-32fea4e50693-serviceca\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:56:24.194646 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.194631 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-daemon-config\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.194704 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.194653 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovnkube-script-lib\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.195891 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.195877 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6533ee11-4dc9-4c0f-b035-1200f74d869b-iptables-alerter-script\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:56:24.202312 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.202299 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 11:56:24.202396 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.202385 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 11:56:24.202396 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.202389 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 11:56:24.209820 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.209794 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bctf\" (UniqueName: \"kubernetes.io/projected/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-kube-api-access-6bctf\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.209998 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.209980 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75mwz\" (UniqueName: \"kubernetes.io/projected/7f547b23-0439-4cc3-9127-387407a34b31-kube-api-access-75mwz\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:56:24.210073 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.210057 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtqw\" (UniqueName: \"kubernetes.io/projected/7e55d04a-613c-4e3a-bcc5-32fea4e50693-kube-api-access-djtqw\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:56:24.210322 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.210304 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhlnz\" (UniqueName: \"kubernetes.io/projected/44d668e0-e534-4e0b-8d32-31ead2f678a2-kube-api-access-hhlnz\") pod \"network-check-target-s2fqn\" (UID: \"44d668e0-e534-4e0b-8d32-31ead2f678a2\") " pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:24.210390 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.210323 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhq46\" (UniqueName: \"kubernetes.io/projected/6533ee11-4dc9-4c0f-b035-1200f74d869b-kube-api-access-hhq46\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:56:24.210390 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.210361 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbbqq\" (UniqueName: \"kubernetes.io/projected/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-kube-api-access-jbbqq\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.214909 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.214891 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8686b\" (UniqueName: \"kubernetes.io/projected/05fb7da7-c7b3-412e-a14e-45d7c10d4216-kube-api-access-8686b\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:24.215021 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.215004 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqh2j\" (UniqueName: \"kubernetes.io/projected/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-kube-api-access-xqh2j\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.215318 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.215300 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5wc6\" (UniqueName: \"kubernetes.io/projected/291cad74-3a77-43f6-abb4-d28aaa5eb20c-kube-api-access-c5wc6\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.215446 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.215430 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kc7z\" (UniqueName: \"kubernetes.io/projected/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kube-api-access-8kc7z\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.248250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248229 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-system-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.248346 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248257 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-sys\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.248346 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248278 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-system-cni-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.248346 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248301 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-slash\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.248346 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248322 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-systemd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.248346 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248339 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-system-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248349 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-sys\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248363 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-slash\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248367 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-systemd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248352 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-system-cni-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248408 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-etc-kubernetes\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248426 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-systemd-units\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248443 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-etc-kubernetes\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248444 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248468 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248477 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248492 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248510 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-systemd-units\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248522 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-sys-fs\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248552 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248556 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-hostroot\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248572 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-var-lib-kubelet\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248575 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-sys-fs\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.248578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248587 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-etc-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248587 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-hostroot\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248604 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysconfig\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248622 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-socket-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248624 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-var-lib-kubelet\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248629 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-etc-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248637 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cnibin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248670 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-cnibin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248673 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysconfig\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248667 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248696 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248718 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-registration-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248675 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-socket-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248743 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248775 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-k8s-cni-cncf-io\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248793 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-cni-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248795 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-netns\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248813 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-netns\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248777 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-registration-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248829 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248797 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-k8s-cni-cncf-io\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248855 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-device-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248875 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248877 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-multus-certs\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248885 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-device-dir\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248903 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248925 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-lib-modules\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248930 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-run-multus-certs\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248951 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-kubelet\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248975 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-var-lib-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.248996 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-modprobe-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249012 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249016 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-lib-modules\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249025 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cnibin\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249045 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-kubelet\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249070 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-var-lib-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249104 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-multus\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249119 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-modprobe-d\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249129 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/291cad74-3a77-43f6-abb4-d28aaa5eb20c-cnibin\") pod \"multus-additional-cni-plugins-mmzdg\" (UID: \"291cad74-3a77-43f6-abb4-d28aaa5eb20c\") " pod="openshift-multus/multus-additional-cni-plugins-mmzdg" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249146 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-systemd\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249183 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-bin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249186 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-systemd\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249180 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-multus\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249197 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-kubelet\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249216 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-kubelet\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249220 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-node-log\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249237 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-kubelet-config\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249250 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e55d04a-613c-4e3a-bcc5-32fea4e50693-host\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249264 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-netd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249268 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-node-log\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.249250 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249278 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-dbus\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249204 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-host-var-lib-cni-bin\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249300 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e55d04a-613c-4e3a-bcc5-32fea4e50693-host\") pod \"node-ca-np85r\" (UID: \"7e55d04a-613c-4e3a-bcc5-32fea4e50693\") " pod="openshift-image-registry/node-ca-np85r" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249309 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-kubelet-config\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249311 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-netd\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249328 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6533ee11-4dc9-4c0f-b035-1200f74d869b-host-slash\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249348 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f547b23-0439-4cc3-9127-387407a34b31-hosts-file\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249349 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/837a6c92-ea3e-490a-810e-05e2fc91a89a-dbus\") pod \"global-pull-secret-syncer-d5dpl\" (UID: \"837a6c92-ea3e-490a-810e-05e2fc91a89a\") " pod="kube-system/global-pull-secret-syncer-d5dpl" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249379 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6533ee11-4dc9-4c0f-b035-1200f74d869b-host-slash\") pod \"iptables-alerter-4ncdp\" (UID: \"6533ee11-4dc9-4c0f-b035-1200f74d869b\") " pod="openshift-network-operator/iptables-alerter-4ncdp" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249371 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-conf\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249409 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249411 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f547b23-0439-4cc3-9127-387407a34b31-hosts-file\") pod \"node-resolver-nnpfg\" (UID: \"7f547b23-0439-4cc3-9127-387407a34b31\") " pod="openshift-dns/node-resolver-nnpfg" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249425 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-conf-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249447 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-log-socket\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249462 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-kubernetes\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249461 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1bcac010-ce3f-4fac-a5a0-5833466e4bd8-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ct42\" (UID: \"1bcac010-ce3f-4fac-a5a0-5833466e4bd8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ct42" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249477 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249479 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-conf-dir\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249462 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-sysctl-conf\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249519 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-log-socket\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249526 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-openvswitch\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249526 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-etc-kubernetes\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249541 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-ovn\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249557 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-bin\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249573 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-run\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249594 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-socket-dir-parent\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249606 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-run-ovn\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249611 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-cni-bin\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249619 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-netns\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249641 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-host\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249662 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bebca75-b1fd-4e59-9a4b-d6589df81d9b-multus-socket-dir-parent\") pod \"multus-w4h5q\" (UID: \"5bebca75-b1fd-4e59-9a4b-d6589df81d9b\") " pod="openshift-multus/multus-w4h5q" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249621 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-run\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249678 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-host-run-netns\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.250732 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.249693 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/59ecdb0c-5e72-4d63-b737-a9c222dc8d2b-host\") pod \"tuned-gnt4x\" (UID: \"59ecdb0c-5e72-4d63-b737-a9c222dc8d2b\") " pod="openshift-cluster-node-tuning-operator/tuned-gnt4x" Nov 02 11:56:24.282708 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.282665 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 11:56:24.282954 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.282935 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x7vnq\"" Nov 02 11:56:24.283050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.283041 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2rrmx\"" Nov 02 11:56:24.283109 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.283067 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4pvvq\"" Nov 02 11:56:24.283109 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.283041 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bjwgn\"" Nov 02 11:56:24.283109 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.283056 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 11:56:24.283109 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.283082 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lk8z8\"" Nov 02 11:56:24.283288 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.283274 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 11:56:24.283339 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.283305 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q864t\"" Nov 02 11:56:24.283383 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.283336 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gpdd6\"" Nov 02 11:56:24.283421 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.283390 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 11:56:24.283421 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.283419 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-g4drg\"" Nov 02 11:56:24.287799 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.287783 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fcda3ab4-1f37-49db-8eb9-2b83fe89d484-agent-certs\") pod \"konnectivity-agent-5pnd2\" (UID: \"fcda3ab4-1f37-49db-8eb9-2b83fe89d484\") " pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:56:24.287871 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.287817 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c5d3609-6aef-4c1f-a876-6cb15457cc9a-ovn-node-metrics-cert\") pod \"ovnkube-node-2wrsc\" (UID: \"1c5d3609-6aef-4c1f-a876-6cb15457cc9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:24.288987 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.288964 4312 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 11:56:24.290907 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.290894 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fb7da7-c7b3-412e-a14e-45d7c10d4216-metrics-certs\") pod \"network-metrics-daemon-p2frl\" (UID: \"05fb7da7-c7b3-412e-a14e-45d7c10d4216\") " pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:24.395356 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.395330 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:24.431215 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.431199 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p2frl" Nov 02 11:56:24.645658 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.645635 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:24.842646 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.842477 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p2frl"] Nov 02 11:56:24.846438 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:24.846406 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fb7da7_c7b3_412e_a14e_45d7c10d4216.slice/crio-7823a4421d1f181202d5be336cd1f65898e7442cc5431327ec22bd103718bdb6 WatchSource:0}: Error finding container 7823a4421d1f181202d5be336cd1f65898e7442cc5431327ec22bd103718bdb6: Status 404 returned error can't find the container with id 7823a4421d1f181202d5be336cd1f65898e7442cc5431327ec22bd103718bdb6 Nov 02 11:56:24.848788 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.848768 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 11:56:24.890523 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:24.890445 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:25.103111 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:25.103076 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2frl" event={"ID":"05fb7da7-c7b3-412e-a14e-45d7c10d4216","Type":"ContainerStarted","Data":"7823a4421d1f181202d5be336cd1f65898e7442cc5431327ec22bd103718bdb6"} Nov 02 11:56:25.862059 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:25.862027 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:26.107214 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:26.107182 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:26.353947 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:26.353914 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:26.602206 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:26.602180 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:26.814019 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:26.813997 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:26.853425 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:26.853395 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:27.109312 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:27.109280 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2frl" event={"ID":"05fb7da7-c7b3-412e-a14e-45d7c10d4216","Type":"ContainerStarted","Data":"20d5b7e963ee7fb0b6f7e33da096410754e3ca61b025549eb899c518ffb49519"} Nov 02 11:56:27.109784 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:27.109331 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p2frl" event={"ID":"05fb7da7-c7b3-412e-a14e-45d7c10d4216","Type":"ContainerStarted","Data":"07cb8aded5bdda34a6b3cc6eb2fd59d990aa91564bb7c034fb8066390793e43d"} Nov 02 11:56:27.111629 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:27.111612 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:27.147195 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:27.147168 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:27.392205 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:27.392141 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:27.641186 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:27.641156 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:27.890259 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:27.890233 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:28.137728 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:28.137698 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:28.391862 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:28.391839 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:28.636259 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:28.636229 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:28.883912 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:28.883887 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:29.263872 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:29.263837 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:29.508428 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:29.508394 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:30.695434 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:30.695398 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:30.695992 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:30.695650 4312 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 11:56:32.035032 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.035005 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5"] Nov 02 11:56:32.036935 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.036915 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5" Nov 02 11:56:32.040728 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.040704 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-rqnr5"] Nov 02 11:56:32.042366 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.042346 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bswsk"] Nov 02 11:56:32.042524 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.042482 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-rqnr5" Nov 02 11:56:32.044804 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.044789 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.084509 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.084480 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p7qcg"] Nov 02 11:56:32.086181 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.086167 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.089679 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.089663 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 11:56:32.090068 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.090053 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h92tx\"" Nov 02 11:56:32.095390 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.095370 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4h4z\" (UniqueName: \"kubernetes.io/projected/9dab2f42-5476-4188-85f9-68f15624824c-kube-api-access-c4h4z\") pod \"downloads-6dd789c8b7-rqnr5\" (UID: \"9dab2f42-5476-4188-85f9-68f15624824c\") " pod="openshift-console/downloads-6dd789c8b7-rqnr5" Nov 02 11:56:32.095474 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.095396 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dab2f42-5476-4188-85f9-68f15624824c-tmp\") pod \"downloads-6dd789c8b7-rqnr5\" (UID: \"9dab2f42-5476-4188-85f9-68f15624824c\") " pod="openshift-console/downloads-6dd789c8b7-rqnr5" Nov 02 11:56:32.095474 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.095412 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/113706d4-49f7-4cdd-87d3-d78bdc0c273b-config-volume\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.095581 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.095474 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/113706d4-49f7-4cdd-87d3-d78bdc0c273b-metrics-tls\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.095616 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.095583 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcptg\" (UniqueName: \"kubernetes.io/projected/113706d4-49f7-4cdd-87d3-d78bdc0c273b-kube-api-access-rcptg\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.095616 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.095610 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/af313ed2-a9bc-4b05-a16c-5b0280657674-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f72f5\" (UID: \"af313ed2-a9bc-4b05-a16c-5b0280657674\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5" Nov 02 11:56:32.095671 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.095632 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113706d4-49f7-4cdd-87d3-d78bdc0c273b-tmp-dir\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.098787 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.098772 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 11:56:32.106628 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.106588 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5"] Nov 02 11:56:32.110220 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.110197 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n6c2k"] Nov 02 11:56:32.111957 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.111935 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n6c2k" Nov 02 11:56:32.114951 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.114684 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86c65f44df-65bds"] Nov 02 11:56:32.116300 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.116284 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.139202 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.139185 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 11:56:32.139316 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.139262 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 11:56:32.139316 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.139287 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 11:56:32.139316 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.139306 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 11:56:32.139316 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.139307 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 11:56:32.139763 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.139746 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-rqnr5"] Nov 02 11:56:32.140138 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.140124 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bswsk"] Nov 02 11:56:32.164757 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.164737 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 11:56:32.164898 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.164882 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 11:56:32.179684 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.179664 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n6c2k"] Nov 02 11:56:32.181430 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.181349 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86c65f44df-65bds"] Nov 02 11:56:32.188241 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.188214 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p7qcg"] Nov 02 11:56:32.190390 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.190370 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qcv25\"" Nov 02 11:56:32.190514 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.190370 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 11:56:32.190578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.190558 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 11:56:32.190578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.190571 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hqqr9\"" Nov 02 11:56:32.190687 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.190648 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7p526\"" Nov 02 11:56:32.190779 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.190760 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 11:56:32.190858 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.190802 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-bcp4s\"" Nov 02 11:56:32.190858 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.190815 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 11:56:32.190858 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.190830 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 11:56:32.195905 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.195883 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b304d901-cfcd-464f-a683-5e1c60359bf0-image-registry-private-configuration\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.196008 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.195924 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b38dd5-0616-4a37-8a51-eb6eab1f6357-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.196008 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.195951 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b304d901-cfcd-464f-a683-5e1c60359bf0-ca-trust-extracted\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.196008 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.195980 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwb2v\" (UniqueName: \"kubernetes.io/projected/b304d901-cfcd-464f-a683-5e1c60359bf0-kube-api-access-rwb2v\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.196168 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196065 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/113706d4-49f7-4cdd-87d3-d78bdc0c273b-metrics-tls\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.196168 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196093 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7t7c\" (UniqueName: \"kubernetes.io/projected/a63f9756-2066-4880-8552-088c285a85b9-kube-api-access-x7t7c\") pod \"ingress-canary-n6c2k\" (UID: \"a63f9756-2066-4880-8552-088c285a85b9\") " pod="openshift-ingress-canary/ingress-canary-n6c2k" Nov 02 11:56:32.196168 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196123 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcptg\" (UniqueName: \"kubernetes.io/projected/113706d4-49f7-4cdd-87d3-d78bdc0c273b-kube-api-access-rcptg\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.196342 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196171 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/af313ed2-a9bc-4b05-a16c-5b0280657674-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f72f5\" (UID: \"af313ed2-a9bc-4b05-a16c-5b0280657674\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5" Nov 02 11:56:32.196342 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196206 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a63f9756-2066-4880-8552-088c285a85b9-cert\") pod \"ingress-canary-n6c2k\" (UID: \"a63f9756-2066-4880-8552-088c285a85b9\") " pod="openshift-ingress-canary/ingress-canary-n6c2k" Nov 02 11:56:32.196342 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196315 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23b38dd5-0616-4a37-8a51-eb6eab1f6357-data-volume\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.196463 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196345 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b304d901-cfcd-464f-a683-5e1c60359bf0-registry-certificates\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.196554 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196538 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113706d4-49f7-4cdd-87d3-d78bdc0c273b-tmp-dir\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.196590 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196562 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23b38dd5-0616-4a37-8a51-eb6eab1f6357-crio-socket\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.196590 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196577 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b304d901-cfcd-464f-a683-5e1c60359bf0-trusted-ca\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.196672 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196598 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4h4z\" (UniqueName: \"kubernetes.io/projected/9dab2f42-5476-4188-85f9-68f15624824c-kube-api-access-c4h4z\") pod \"downloads-6dd789c8b7-rqnr5\" (UID: \"9dab2f42-5476-4188-85f9-68f15624824c\") " pod="openshift-console/downloads-6dd789c8b7-rqnr5" Nov 02 11:56:32.196672 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196618 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dab2f42-5476-4188-85f9-68f15624824c-tmp\") pod \"downloads-6dd789c8b7-rqnr5\" (UID: \"9dab2f42-5476-4188-85f9-68f15624824c\") " pod="openshift-console/downloads-6dd789c8b7-rqnr5" Nov 02 11:56:32.196758 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196676 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/113706d4-49f7-4cdd-87d3-d78bdc0c273b-config-volume\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.196758 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196735 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28q76\" (UniqueName: \"kubernetes.io/projected/23b38dd5-0616-4a37-8a51-eb6eab1f6357-kube-api-access-28q76\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.196862 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196761 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b304d901-cfcd-464f-a683-5e1c60359bf0-installation-pull-secrets\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.196862 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196789 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b38dd5-0616-4a37-8a51-eb6eab1f6357-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.196970 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196857 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113706d4-49f7-4cdd-87d3-d78bdc0c273b-tmp-dir\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.196970 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196862 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b304d901-cfcd-464f-a683-5e1c60359bf0-registry-tls\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.196970 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.196909 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b304d901-cfcd-464f-a683-5e1c60359bf0-bound-sa-token\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.197134 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.197112 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9dab2f42-5476-4188-85f9-68f15624824c-tmp\") pod \"downloads-6dd789c8b7-rqnr5\" (UID: \"9dab2f42-5476-4188-85f9-68f15624824c\") " pod="openshift-console/downloads-6dd789c8b7-rqnr5" Nov 02 11:56:32.197328 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.197312 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/113706d4-49f7-4cdd-87d3-d78bdc0c273b-config-volume\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.198574 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.198554 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/113706d4-49f7-4cdd-87d3-d78bdc0c273b-metrics-tls\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.198664 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.198651 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/af313ed2-a9bc-4b05-a16c-5b0280657674-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f72f5\" (UID: \"af313ed2-a9bc-4b05-a16c-5b0280657674\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5" Nov 02 11:56:32.199797 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.199784 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4wp8v\"" Nov 02 11:56:32.199839 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.199823 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 11:56:32.240165 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.240138 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 11:56:32.256547 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.256522 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4h4z\" (UniqueName: \"kubernetes.io/projected/9dab2f42-5476-4188-85f9-68f15624824c-kube-api-access-c4h4z\") pod \"downloads-6dd789c8b7-rqnr5\" (UID: \"9dab2f42-5476-4188-85f9-68f15624824c\") " pod="openshift-console/downloads-6dd789c8b7-rqnr5" Nov 02 11:56:32.256547 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.256536 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcptg\" (UniqueName: \"kubernetes.io/projected/113706d4-49f7-4cdd-87d3-d78bdc0c273b-kube-api-access-rcptg\") pod \"dns-default-bswsk\" (UID: \"113706d4-49f7-4cdd-87d3-d78bdc0c273b\") " pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.297662 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.297610 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7t7c\" (UniqueName: \"kubernetes.io/projected/a63f9756-2066-4880-8552-088c285a85b9-kube-api-access-x7t7c\") pod \"ingress-canary-n6c2k\" (UID: \"a63f9756-2066-4880-8552-088c285a85b9\") " pod="openshift-ingress-canary/ingress-canary-n6c2k" Nov 02 11:56:32.297662 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.297644 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a63f9756-2066-4880-8552-088c285a85b9-cert\") pod \"ingress-canary-n6c2k\" (UID: \"a63f9756-2066-4880-8552-088c285a85b9\") " pod="openshift-ingress-canary/ingress-canary-n6c2k" Nov 02 11:56:32.297791 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.297667 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23b38dd5-0616-4a37-8a51-eb6eab1f6357-data-volume\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.297791 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.297690 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b304d901-cfcd-464f-a683-5e1c60359bf0-registry-certificates\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.297791 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.297720 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23b38dd5-0616-4a37-8a51-eb6eab1f6357-crio-socket\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.297791 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.297741 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b304d901-cfcd-464f-a683-5e1c60359bf0-trusted-ca\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.297988 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.297832 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23b38dd5-0616-4a37-8a51-eb6eab1f6357-crio-socket\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.297988 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.297847 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28q76\" (UniqueName: \"kubernetes.io/projected/23b38dd5-0616-4a37-8a51-eb6eab1f6357-kube-api-access-28q76\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.297988 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.297898 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b304d901-cfcd-464f-a683-5e1c60359bf0-installation-pull-secrets\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.297988 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.297946 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23b38dd5-0616-4a37-8a51-eb6eab1f6357-data-volume\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.297988 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.297934 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b38dd5-0616-4a37-8a51-eb6eab1f6357-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.298236 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.298001 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b304d901-cfcd-464f-a683-5e1c60359bf0-registry-tls\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.298236 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.298030 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b304d901-cfcd-464f-a683-5e1c60359bf0-bound-sa-token\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.298236 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.298071 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b304d901-cfcd-464f-a683-5e1c60359bf0-image-registry-private-configuration\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.298236 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.298102 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b38dd5-0616-4a37-8a51-eb6eab1f6357-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.298236 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.298134 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b304d901-cfcd-464f-a683-5e1c60359bf0-ca-trust-extracted\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.298236 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.298168 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwb2v\" (UniqueName: \"kubernetes.io/projected/b304d901-cfcd-464f-a683-5e1c60359bf0-kube-api-access-rwb2v\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.298571 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.298486 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23b38dd5-0616-4a37-8a51-eb6eab1f6357-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.298636 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.298607 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b304d901-cfcd-464f-a683-5e1c60359bf0-registry-certificates\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.298820 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.298794 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b304d901-cfcd-464f-a683-5e1c60359bf0-ca-trust-extracted\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.298944 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.298925 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b304d901-cfcd-464f-a683-5e1c60359bf0-trusted-ca\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.300007 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.299983 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a63f9756-2066-4880-8552-088c285a85b9-cert\") pod \"ingress-canary-n6c2k\" (UID: \"a63f9756-2066-4880-8552-088c285a85b9\") " pod="openshift-ingress-canary/ingress-canary-n6c2k" Nov 02 11:56:32.300408 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.300390 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23b38dd5-0616-4a37-8a51-eb6eab1f6357-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.300589 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.300569 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b304d901-cfcd-464f-a683-5e1c60359bf0-installation-pull-secrets\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.300756 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.300737 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b304d901-cfcd-464f-a683-5e1c60359bf0-registry-tls\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.300921 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.300901 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b304d901-cfcd-464f-a683-5e1c60359bf0-image-registry-private-configuration\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.308595 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.308575 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:32.348761 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.348746 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5" Nov 02 11:56:32.354511 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.354477 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-rqnr5" Nov 02 11:56:32.358349 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.358328 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b304d901-cfcd-464f-a683-5e1c60359bf0-bound-sa-token\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.358766 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.358747 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28q76\" (UniqueName: \"kubernetes.io/projected/23b38dd5-0616-4a37-8a51-eb6eab1f6357-kube-api-access-28q76\") pod \"insights-runtime-extractor-p7qcg\" (UID: \"23b38dd5-0616-4a37-8a51-eb6eab1f6357\") " pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.359248 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.359222 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7t7c\" (UniqueName: \"kubernetes.io/projected/a63f9756-2066-4880-8552-088c285a85b9-kube-api-access-x7t7c\") pod \"ingress-canary-n6c2k\" (UID: \"a63f9756-2066-4880-8552-088c285a85b9\") " pod="openshift-ingress-canary/ingress-canary-n6c2k" Nov 02 11:56:32.359994 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.359971 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwb2v\" (UniqueName: \"kubernetes.io/projected/b304d901-cfcd-464f-a683-5e1c60359bf0-kube-api-access-rwb2v\") pod \"image-registry-86c65f44df-65bds\" (UID: \"b304d901-cfcd-464f-a683-5e1c60359bf0\") " pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.359994 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.359991 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:32.397750 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.397536 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p7qcg" Nov 02 11:56:32.426887 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.426662 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n6c2k" Nov 02 11:56:32.431026 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.430666 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:32.453422 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.453398 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2wrsc" Nov 02 11:56:32.559931 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.559870 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:32.717373 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.717348 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5"] Nov 02 11:56:32.720385 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:32.720363 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf313ed2_a9bc_4b05_a16c_5b0280657674.slice/crio-154917bfa195f8ea8ad90fb9db07e413ccc6980431456c35e5b67d43698c846f WatchSource:0}: Error finding container 154917bfa195f8ea8ad90fb9db07e413ccc6980431456c35e5b67d43698c846f: Status 404 returned error can't find the container with id 154917bfa195f8ea8ad90fb9db07e413ccc6980431456c35e5b67d43698c846f Nov 02 11:56:32.776825 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.775706 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-rqnr5"] Nov 02 11:56:32.776825 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.775813 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bswsk"] Nov 02 11:56:32.812820 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.812762 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:32.839726 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.839706 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n6c2k"] Nov 02 11:56:32.842818 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:32.842792 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63f9756_2066_4880_8552_088c285a85b9.slice/crio-e79aaf8fcaf220723a90db659fce858bab0ab37d8acf507cad63cbe7ae4bd029 WatchSource:0}: Error finding container e79aaf8fcaf220723a90db659fce858bab0ab37d8acf507cad63cbe7ae4bd029: Status 404 returned error can't find the container with id e79aaf8fcaf220723a90db659fce858bab0ab37d8acf507cad63cbe7ae4bd029 Nov 02 11:56:32.847523 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.847323 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86c65f44df-65bds"] Nov 02 11:56:32.850125 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:32.850101 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb304d901_cfcd_464f_a683_5e1c60359bf0.slice/crio-cb2e0cddc05e0e543c28bb3b2b0934ffa6fe7800b59b0e44044813ac5a05bfb0 WatchSource:0}: Error finding container cb2e0cddc05e0e543c28bb3b2b0934ffa6fe7800b59b0e44044813ac5a05bfb0: Status 404 returned error can't find the container with id cb2e0cddc05e0e543c28bb3b2b0934ffa6fe7800b59b0e44044813ac5a05bfb0 Nov 02 11:56:32.939404 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:32.939379 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p7qcg"] Nov 02 11:56:32.942177 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:32.942154 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b38dd5_0616_4a37_8a51_eb6eab1f6357.slice/crio-3cd4c56a72dd4612c962e3c89ebce98fdd8e1d8995b68a061507278d298e3db5 WatchSource:0}: Error finding container 3cd4c56a72dd4612c962e3c89ebce98fdd8e1d8995b68a061507278d298e3db5: Status 404 returned error can't find the container with id 3cd4c56a72dd4612c962e3c89ebce98fdd8e1d8995b68a061507278d298e3db5 Nov 02 11:56:33.060900 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:33.060873 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:33.124537 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:33.124456 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5" event={"ID":"af313ed2-a9bc-4b05-a16c-5b0280657674","Type":"ContainerStarted","Data":"154917bfa195f8ea8ad90fb9db07e413ccc6980431456c35e5b67d43698c846f"} Nov 02 11:56:33.125752 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:33.125723 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n6c2k" event={"ID":"a63f9756-2066-4880-8552-088c285a85b9","Type":"ContainerStarted","Data":"e79aaf8fcaf220723a90db659fce858bab0ab37d8acf507cad63cbe7ae4bd029"} Nov 02 11:56:33.126927 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:33.126894 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bswsk" event={"ID":"113706d4-49f7-4cdd-87d3-d78bdc0c273b","Type":"ContainerStarted","Data":"45b3abf2346dbda5b5d1c84ed071f0fb1f4305af19781b6ebbbcba0d258126ad"} Nov 02 11:56:33.128147 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:33.128113 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-rqnr5" event={"ID":"9dab2f42-5476-4188-85f9-68f15624824c","Type":"ContainerStarted","Data":"4179419835e5a77a2b46525f60b8b759a67aa1439a339225af1af7295983243a"} Nov 02 11:56:33.129690 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:33.129668 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86c65f44df-65bds" event={"ID":"b304d901-cfcd-464f-a683-5e1c60359bf0","Type":"ContainerStarted","Data":"d27fec0956d2a3db017b53ad1194be4a4a606754fa3fa51eb6a11d54e908d5d2"} Nov 02 11:56:33.129784 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:33.129707 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86c65f44df-65bds" event={"ID":"b304d901-cfcd-464f-a683-5e1c60359bf0","Type":"ContainerStarted","Data":"cb2e0cddc05e0e543c28bb3b2b0934ffa6fe7800b59b0e44044813ac5a05bfb0"} Nov 02 11:56:33.131170 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:33.131146 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7qcg" event={"ID":"23b38dd5-0616-4a37-8a51-eb6eab1f6357","Type":"ContainerStarted","Data":"b213e9f53b43ae4a29367731cf2cb8b17ccefec7860ee7843eeb6e38fd077a3c"} Nov 02 11:56:33.131249 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:33.131185 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7qcg" event={"ID":"23b38dd5-0616-4a37-8a51-eb6eab1f6357","Type":"ContainerStarted","Data":"3cd4c56a72dd4612c962e3c89ebce98fdd8e1d8995b68a061507278d298e3db5"} Nov 02 11:56:33.250056 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:33.250005 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86c65f44df-65bds" podStartSLOduration=1.249991582 podStartE2EDuration="1.249991582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:33.24945712 +0000 UTC m=+10.279693370" watchObservedRunningTime="2026-11-02 11:56:33.249991582 +0000 UTC m=+10.280227823" Nov 02 11:56:34.136076 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:34.136042 4312 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 11:56:35.142177 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:35.142142 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5" event={"ID":"af313ed2-a9bc-4b05-a16c-5b0280657674","Type":"ContainerStarted","Data":"7a8420fb2306545f5ee0d461b3a16f3196eddc901dd3983ba99de625a463255a"} Nov 02 11:56:35.145395 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:35.145310 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7qcg" event={"ID":"23b38dd5-0616-4a37-8a51-eb6eab1f6357","Type":"ContainerStarted","Data":"2d399dd4609aadaa40537f38347f24b8cecbdb1b51f0fcd864de29af541196b6"} Nov 02 11:56:35.258076 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:35.258027 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5" podStartSLOduration=2.383586435 podStartE2EDuration="4.258014345s" totalImagesPullingTime="1.87442791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:35.257561266 +0000 UTC m=+12.287797519" watchObservedRunningTime="2026-11-02 11:56:35.258014345 +0000 UTC m=+12.288250595" Nov 02 11:56:35.942012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:35.941978 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:36.152469 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.152439 4312 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 11:56:36.152469 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.152438 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bswsk" event={"ID":"113706d4-49f7-4cdd-87d3-d78bdc0c273b","Type":"ContainerStarted","Data":"59aaf5483a6be01713cf05c0862f4ae9cea533c3d8aa09dd79aa0394850858d9"} Nov 02 11:56:36.153018 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.152527 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bswsk" event={"ID":"113706d4-49f7-4cdd-87d3-d78bdc0c273b","Type":"ContainerStarted","Data":"5daca6fe5655eef4167ee26733740b6e2cb65b904aeece0b7d06d7ccabe76bf2"} Nov 02 11:56:36.153018 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.152783 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:36.184737 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.184708 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:36.256535 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.256448 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-64796bd696-rnrjq"] Nov 02 11:56:36.258572 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.258539 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.306119 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.306061 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-bswsk" podStartSLOduration=2.569924216 podStartE2EDuration="5.306044182s" totalImagesPullingTime="2.736119966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:36.270579322 +0000 UTC m=+13.300815572" watchObservedRunningTime="2026-11-02 11:56:36.306044182 +0000 UTC m=+13.336280410" Nov 02 11:56:36.307177 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.307151 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64796bd696-rnrjq"] Nov 02 11:56:36.319940 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.319892 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 11:56:36.320293 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.320246 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 11:56:36.320293 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.320298 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 11:56:36.320448 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.320381 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 11:56:36.320448 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.320397 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 11:56:36.320561 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.320484 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kvndk\"" Nov 02 11:56:36.335891 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.335871 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-oauth-config\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.335998 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.335904 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-oauth-serving-cert\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.335998 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.335939 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-config\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.335998 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.335971 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-serving-cert\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.336146 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.336007 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2mwk\" (UniqueName: \"kubernetes.io/projected/2c8a1f9b-6089-4841-a311-dd0df9efe691-kube-api-access-f2mwk\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.336146 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.336035 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c8a1f9b-6089-4841-a311-dd0df9efe691-tmp\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.336146 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.336066 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-service-ca\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.439050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.437284 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-oauth-config\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.439050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.437324 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-oauth-serving-cert\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.439050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.437352 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-config\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.439050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.437381 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-serving-cert\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.439050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.437423 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2mwk\" (UniqueName: \"kubernetes.io/projected/2c8a1f9b-6089-4841-a311-dd0df9efe691-kube-api-access-f2mwk\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.439050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.437452 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c8a1f9b-6089-4841-a311-dd0df9efe691-tmp\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.439050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.437475 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-service-ca\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.439050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.438318 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-service-ca\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.439050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.438398 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c8a1f9b-6089-4841-a311-dd0df9efe691-tmp\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.439050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.438430 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-oauth-serving-cert\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.439050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.439006 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-config\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.440222 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.440196 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-serving-cert\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.440891 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.440870 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-oauth-config\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.506206 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.499559 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2mwk\" (UniqueName: \"kubernetes.io/projected/2c8a1f9b-6089-4841-a311-dd0df9efe691-kube-api-access-f2mwk\") pod \"console-64796bd696-rnrjq\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.571805 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.571543 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:36.949871 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:36.949838 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64796bd696-rnrjq"] Nov 02 11:56:36.953173 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:36.953128 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8a1f9b_6089_4841_a311_dd0df9efe691.slice/crio-94910d01d0d579fe026bc52717a2e58c587b3f8ab949ccfcd5fc30369cd29c1c WatchSource:0}: Error finding container 94910d01d0d579fe026bc52717a2e58c587b3f8ab949ccfcd5fc30369cd29c1c: Status 404 returned error can't find the container with id 94910d01d0d579fe026bc52717a2e58c587b3f8ab949ccfcd5fc30369cd29c1c Nov 02 11:56:37.157295 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:37.157255 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n6c2k" event={"ID":"a63f9756-2066-4880-8552-088c285a85b9","Type":"ContainerStarted","Data":"e4e3896623b0c491ecbe01ef7fe83f0397bc5ea29c1273218165d0c37dd8ba90"} Nov 02 11:56:37.159374 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:37.159317 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64796bd696-rnrjq" event={"ID":"2c8a1f9b-6089-4841-a311-dd0df9efe691","Type":"ContainerStarted","Data":"94910d01d0d579fe026bc52717a2e58c587b3f8ab949ccfcd5fc30369cd29c1c"} Nov 02 11:56:37.161828 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:37.161794 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p7qcg" event={"ID":"23b38dd5-0616-4a37-8a51-eb6eab1f6357","Type":"ContainerStarted","Data":"56eeca1f22c1b59568b9ea1ce1ff6f865c55d3ef5f5e3546576ff4d2e0f7008c"} Nov 02 11:56:37.180414 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:37.180393 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5" Nov 02 11:56:37.180536 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:37.180468 4312 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 11:56:37.185030 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:37.185013 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f72f5" Nov 02 11:56:37.273841 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:37.273664 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n6c2k" podStartSLOduration=1.49407604 podStartE2EDuration="5.27364983s" totalImagesPullingTime="3.77957379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:37.272660244 +0000 UTC m=+14.302896496" watchObservedRunningTime="2026-11-02 11:56:37.27364983 +0000 UTC m=+14.303886080" Nov 02 11:56:37.504767 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:37.504717 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p7qcg" podStartSLOduration=3.081295685 podStartE2EDuration="6.504704939s" totalImagesPullingTime="3.423409254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 11:56:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:37.503953922 +0000 UTC m=+14.534190174" watchObservedRunningTime="2026-11-02 11:56:37.504704939 +0000 UTC m=+14.534941231" Nov 02 11:56:37.865677 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:37.865648 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:38.108882 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.108843 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:38.167568 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.167541 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt"] Nov 02 11:56:38.169575 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.169552 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.221901 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.221879 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 11:56:38.222006 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.221926 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 11:56:38.253431 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.253399 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52c28f9-63be-48b7-9643-fac197099856-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.253561 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.253454 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rzc\" (UniqueName: \"kubernetes.io/projected/f52c28f9-63be-48b7-9643-fac197099856-kube-api-access-64rzc\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.253561 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.253543 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f52c28f9-63be-48b7-9643-fac197099856-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.253686 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.253580 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f52c28f9-63be-48b7-9643-fac197099856-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.271454 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.271433 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 11:56:38.271577 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.271536 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 11:56:38.271577 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.271564 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 11:56:38.271826 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.271807 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xcqtx\"" Nov 02 11:56:38.272884 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.272866 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt"] Nov 02 11:56:38.354083 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.354051 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rzc\" (UniqueName: \"kubernetes.io/projected/f52c28f9-63be-48b7-9643-fac197099856-kube-api-access-64rzc\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.354237 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.354128 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f52c28f9-63be-48b7-9643-fac197099856-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.354237 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.354175 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f52c28f9-63be-48b7-9643-fac197099856-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.354362 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.354232 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52c28f9-63be-48b7-9643-fac197099856-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.355220 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.355173 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f52c28f9-63be-48b7-9643-fac197099856-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.357582 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.357558 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f52c28f9-63be-48b7-9643-fac197099856-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.357667 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.357564 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f52c28f9-63be-48b7-9643-fac197099856-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.416079 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.416050 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rzc\" (UniqueName: \"kubernetes.io/projected/f52c28f9-63be-48b7-9643-fac197099856-kube-api-access-64rzc\") pod \"prometheus-operator-9cd89b74c-ttsrt\" (UID: \"f52c28f9-63be-48b7-9643-fac197099856\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.483110 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.483042 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" Nov 02 11:56:38.489972 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.489907 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:38.861784 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:38.861703 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt"] Nov 02 11:56:38.906616 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:38.906582 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52c28f9_63be_48b7_9643_fac197099856.slice/crio-756236e495957c3b068dffcb9fdc283742bdaa3ea068f654662205002d8730da WatchSource:0}: Error finding container 756236e495957c3b068dffcb9fdc283742bdaa3ea068f654662205002d8730da: Status 404 returned error can't find the container with id 756236e495957c3b068dffcb9fdc283742bdaa3ea068f654662205002d8730da Nov 02 11:56:39.178490 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:39.178443 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" event={"ID":"f52c28f9-63be-48b7-9643-fac197099856","Type":"ContainerStarted","Data":"756236e495957c3b068dffcb9fdc283742bdaa3ea068f654662205002d8730da"} Nov 02 11:56:40.556827 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:40.556792 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:41.188173 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:41.187748 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64796bd696-rnrjq" event={"ID":"2c8a1f9b-6089-4841-a311-dd0df9efe691","Type":"ContainerStarted","Data":"592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881"} Nov 02 11:56:41.189830 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:41.189804 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" event={"ID":"f52c28f9-63be-48b7-9643-fac197099856","Type":"ContainerStarted","Data":"20b41bb5f7c28b390ba85a9b17c604bc11022037584cb7f25f6da117ff6c4220"} Nov 02 11:56:41.189945 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:41.189855 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" event={"ID":"f52c28f9-63be-48b7-9643-fac197099856","Type":"ContainerStarted","Data":"c024e731e01ed3004b83af885ce8980d5fb8e9fe064c41cabc3847bc5b2ce864"} Nov 02 11:56:41.304974 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:41.304923 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-64796bd696-rnrjq" podStartSLOduration=1.634480376 podStartE2EDuration="5.304911688s" totalImagesPullingTime="3.670431312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:41.304399066 +0000 UTC m=+18.334635316" watchObservedRunningTime="2026-11-02 11:56:41.304911688 +0000 UTC m=+18.335147938" Nov 02 11:56:41.422255 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:41.422202 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-ttsrt" podStartSLOduration=1.681509269 podStartE2EDuration="3.422187121s" totalImagesPullingTime="1.740677852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:41.421560407 +0000 UTC m=+18.451796661" watchObservedRunningTime="2026-11-02 11:56:41.422187121 +0000 UTC m=+18.452423373" Nov 02 11:56:42.672480 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:42.672447 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5pnd2" Nov 02 11:56:43.413742 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:43.413705 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:44.149456 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.149423 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:44.391310 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.391279 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:44.705100 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.705065 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl"] Nov 02 11:56:44.710837 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.710816 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.803081 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.803055 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.803221 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.803091 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdcfd\" (UniqueName: \"kubernetes.io/projected/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-kube-api-access-hdcfd\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.803221 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.803192 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.803369 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.803240 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.812701 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.812676 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 11:56:44.812809 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.812711 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sfxnd\"" Nov 02 11:56:44.812809 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.812741 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 11:56:44.813748 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.813724 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl"] Nov 02 11:56:44.814410 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.814392 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tvcvg"] Nov 02 11:56:44.816426 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.816406 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:44.868473 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.868438 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 11:56:44.868646 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.868612 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-brq6w\"" Nov 02 11:56:44.868742 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.868648 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 11:56:44.868742 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.868689 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 11:56:44.904093 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904070 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.904191 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904103 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7ms9\" (UniqueName: \"kubernetes.io/projected/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-kube-api-access-j7ms9\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:44.904191 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904141 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:44.904191 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904170 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.904312 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904214 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-sys\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:44.904352 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904337 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-wtmp\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:44.904431 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904412 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-root\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:44.904556 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904462 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-textfile\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:44.904556 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904513 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:44.904556 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904536 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.904748 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904558 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdcfd\" (UniqueName: \"kubernetes.io/projected/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-kube-api-access-hdcfd\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.904748 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904592 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-tls\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:44.904748 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904642 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-metrics-client-ca\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:44.904895 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.904869 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.906931 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.906908 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.907024 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.906962 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:44.965102 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:44.965026 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdcfd\" (UniqueName: \"kubernetes.io/projected/bb36a3de-e4d1-4be1-b4c9-608e882fa7d9-kube-api-access-hdcfd\") pod \"openshift-state-metrics-7bd7489957-bw6tl\" (UID: \"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:45.005680 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.005655 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.005800 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.005684 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-sys\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.005800 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.005701 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-wtmp\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.005800 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.005717 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-root\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.005800 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.005743 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-textfile\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.005800 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.005777 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-sys\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.005800 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.005779 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.006080 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.005842 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-tls\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.006080 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.005863 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-wtmp\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.006080 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.005876 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-metrics-client-ca\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.006080 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.005947 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7ms9\" (UniqueName: \"kubernetes.io/projected/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-kube-api-access-j7ms9\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.006268 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.006118 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-root\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.006322 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.006272 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.006398 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.006375 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-textfile\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.006487 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.006379 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-metrics-client-ca\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.008121 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.008096 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.008531 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.008507 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-node-exporter-tls\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.025485 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.025465 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" Nov 02 11:56:45.067511 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.067476 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7ms9\" (UniqueName: \"kubernetes.io/projected/1bb27c13-1527-4670-ae73-b64d4ef2ef0b-kube-api-access-j7ms9\") pod \"node-exporter-tvcvg\" (UID: \"1bb27c13-1527-4670-ae73-b64d4ef2ef0b\") " pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.130094 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.130055 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tvcvg" Nov 02 11:56:45.141395 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:45.141352 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb27c13_1527_4670_ae73_b64d4ef2ef0b.slice/crio-3398f46056188912217cd03dd7253352d1a9e3945978388998746d20fc83d16f WatchSource:0}: Error finding container 3398f46056188912217cd03dd7253352d1a9e3945978388998746d20fc83d16f: Status 404 returned error can't find the container with id 3398f46056188912217cd03dd7253352d1a9e3945978388998746d20fc83d16f Nov 02 11:56:45.208582 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.208515 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvcvg" event={"ID":"1bb27c13-1527-4670-ae73-b64d4ef2ef0b","Type":"ContainerStarted","Data":"3398f46056188912217cd03dd7253352d1a9e3945978388998746d20fc83d16f"} Nov 02 11:56:45.448893 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.448860 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl"] Nov 02 11:56:45.452337 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:45.452305 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb36a3de_e4d1_4be1_b4c9_608e882fa7d9.slice/crio-d222fa41406f79e44538758410adc2253147b739629bba61f853c872dba16124 WatchSource:0}: Error finding container d222fa41406f79e44538758410adc2253147b739629bba61f853c872dba16124: Status 404 returned error can't find the container with id d222fa41406f79e44538758410adc2253147b739629bba61f853c872dba16124 Nov 02 11:56:45.703807 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.703721 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:45.950508 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:45.950466 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:46.164940 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.164912 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bswsk" Nov 02 11:56:46.215369 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.215340 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" event={"ID":"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9","Type":"ContainerStarted","Data":"09d26743e03aa564de8135ceb66a9384d7d654d613fc9843e5c39fda3f7e8878"} Nov 02 11:56:46.215800 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.215400 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" event={"ID":"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9","Type":"ContainerStarted","Data":"655d1fdc06b036391d688db4a9bf599fa0965f5b548afb7e355dc6cd38d3aee5"} Nov 02 11:56:46.215800 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.215425 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" event={"ID":"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9","Type":"ContainerStarted","Data":"d222fa41406f79e44538758410adc2253147b739629bba61f853c872dba16124"} Nov 02 11:56:46.234773 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.234744 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:46.483379 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.483293 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:46.572822 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.572215 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:46.572822 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.572280 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:46.579050 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.579030 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:46.675399 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.675372 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c7779f96f-rpfvd"] Nov 02 11:56:46.677631 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.677606 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.723273 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.723245 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-oauth-serving-cert\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.723399 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.723293 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-oauth-config\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.723399 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.723358 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-config\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.723537 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.723412 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-trusted-ca-bundle\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.723537 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.723438 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-serving-cert\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.723643 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.723611 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-service-ca\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.723685 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.723638 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-tmp\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.723685 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.723665 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzjx7\" (UniqueName: \"kubernetes.io/projected/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-kube-api-access-qzjx7\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.795693 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.795624 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 11:56:46.796707 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.796682 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7779f96f-rpfvd"] Nov 02 11:56:46.824605 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.823979 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-service-ca\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.824605 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.824015 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-tmp\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.824605 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.824036 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjx7\" (UniqueName: \"kubernetes.io/projected/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-kube-api-access-qzjx7\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.824605 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.824064 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-oauth-serving-cert\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.824605 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.824101 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-oauth-config\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.824605 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.824125 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-config\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.824605 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.824178 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-trusted-ca-bundle\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.824605 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.824207 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-serving-cert\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.824605 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.824365 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-tmp\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.825027 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.824780 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-service-ca\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.825306 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.825277 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-oauth-serving-cert\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.825352 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.825320 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-config\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.825721 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.825699 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-trusted-ca-bundle\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.829517 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.827037 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-serving-cert\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.829517 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.827597 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-oauth-config\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.885482 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.885452 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjx7\" (UniqueName: \"kubernetes.io/projected/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-kube-api-access-qzjx7\") pod \"console-c7779f96f-rpfvd\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:46.964731 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.964694 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:46.990779 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:46.990754 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:47.222586 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:47.222544 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" event={"ID":"bb36a3de-e4d1-4be1-b4c9-608e882fa7d9","Type":"ContainerStarted","Data":"2fa51c95b4ad2ff9c72fba5c87cca02c9017ba2d42574b0a55f6f7579c28c005"} Nov 02 11:56:47.224326 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:47.224298 4312 generic.go:356] "Generic (PLEG): container finished" podID="1bb27c13-1527-4670-ae73-b64d4ef2ef0b" containerID="7680d2d8722a196310949bb004bf7dc51a8d1bb917b6d8ac955b69620baf9109" exitCode=0 Nov 02 11:56:47.224451 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:47.224389 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvcvg" event={"ID":"1bb27c13-1527-4670-ae73-b64d4ef2ef0b","Type":"ContainerDied","Data":"7680d2d8722a196310949bb004bf7dc51a8d1bb917b6d8ac955b69620baf9109"} Nov 02 11:56:47.229527 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:47.229505 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:56:47.270582 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:47.270556 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:47.342521 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:47.342455 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bw6tl" podStartSLOduration=1.940572228 podStartE2EDuration="3.342436646s" totalImagesPullingTime="1.401864418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:47.340809918 +0000 UTC m=+24.371046183" watchObservedRunningTime="2026-11-02 11:56:47.342436646 +0000 UTC m=+24.372672897" Nov 02 11:56:47.368907 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:47.368863 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c7779f96f-rpfvd"] Nov 02 11:56:47.396176 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:47.396145 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccaf9ca_fa6b_44de_bafe_4705f66874ed.slice/crio-bb39f5292ccd1f3895882ef14d224f5fd80f2bbd7692ee53f4c0ebd017fafa21 WatchSource:0}: Error finding container bb39f5292ccd1f3895882ef14d224f5fd80f2bbd7692ee53f4c0ebd017fafa21: Status 404 returned error can't find the container with id bb39f5292ccd1f3895882ef14d224f5fd80f2bbd7692ee53f4c0ebd017fafa21 Nov 02 11:56:48.231044 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:48.230469 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvcvg" event={"ID":"1bb27c13-1527-4670-ae73-b64d4ef2ef0b","Type":"ContainerStarted","Data":"ebe981944f18f49b6c2aebd6c2d4aad6f6934d3f8dcada168cd4f4929147ff19"} Nov 02 11:56:48.231044 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:48.230580 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tvcvg" event={"ID":"1bb27c13-1527-4670-ae73-b64d4ef2ef0b","Type":"ContainerStarted","Data":"3c8383e0fb4bdc9a47f9acb5eda7db5985a4be17f2b17cdc3091c0d328ea0e8c"} Nov 02 11:56:48.232772 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:48.232744 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7779f96f-rpfvd" event={"ID":"8ccaf9ca-fa6b-44de-bafe-4705f66874ed","Type":"ContainerStarted","Data":"3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c"} Nov 02 11:56:48.232891 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:48.232792 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7779f96f-rpfvd" event={"ID":"8ccaf9ca-fa6b-44de-bafe-4705f66874ed","Type":"ContainerStarted","Data":"bb39f5292ccd1f3895882ef14d224f5fd80f2bbd7692ee53f4c0ebd017fafa21"} Nov 02 11:56:48.352075 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:48.352014 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tvcvg" podStartSLOduration=2.882671616 podStartE2EDuration="4.35199792s" totalImagesPullingTime="1.387614264s" totalInitContainerRuntime="81.71204ms" isStatefulPod=true podCreationTimestamp="2026-11-02 11:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:48.350602289 +0000 UTC m=+25.380838737" watchObservedRunningTime="2026-11-02 11:56:48.35199792 +0000 UTC m=+25.382234171" Nov 02 11:56:48.470337 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:48.469848 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c7779f96f-rpfvd" podStartSLOduration=2.4698313179999998 podStartE2EDuration="2.469831318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:48.469174762 +0000 UTC m=+25.499411009" watchObservedRunningTime="2026-11-02 11:56:48.469831318 +0000 UTC m=+25.500067570" Nov 02 11:56:48.995281 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:48.995235 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:49.243842 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:49.243808 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:49.491372 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:49.491336 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:49.784474 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:49.784391 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:50.030765 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:50.030734 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:50.561442 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:50.561409 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86c65f44df-65bds" Nov 02 11:56:50.944418 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:50.944393 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:51.196723 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:51.196686 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:51.250792 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:51.250757 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-rqnr5" event={"ID":"9dab2f42-5476-4188-85f9-68f15624824c","Type":"ContainerStarted","Data":"93b527ef4ee12741716c9d97e65dd444b474932e114c38816216e2275fe864bd"} Nov 02 11:56:51.251035 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:51.251018 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-rqnr5" Nov 02 11:56:51.266711 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:51.266685 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-rqnr5" Nov 02 11:56:51.368231 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:51.368176 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-rqnr5" podStartSLOduration=2.261487891 podStartE2EDuration="20.368163545s" totalImagesPullingTime="18.106675654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:51.367878866 +0000 UTC m=+28.398115122" watchObservedRunningTime="2026-11-02 11:56:51.368163545 +0000 UTC m=+28.398399794" Nov 02 11:56:51.441965 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:51.441932 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:53.918310 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.918276 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 11:56:53.944396 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.944369 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.988532 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.988486 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.988698 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.988610 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.988698 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.988644 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4gz6\" (UniqueName: \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-kube-api-access-t4gz6\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.988698 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.988672 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.988822 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.988708 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.988822 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.988776 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.988919 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.988881 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.988919 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.988902 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989003 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.988935 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989003 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.988958 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989003 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.988993 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989125 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.989020 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989125 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.989050 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989125 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.989077 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989125 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.989104 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989486 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.989134 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989486 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.989152 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989486 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.989176 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989486 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.989196 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:53.989486 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.989219 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.001531 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.998326 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 11:56:54.001531 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.998735 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 11:56:54.001531 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.999127 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 11:56:54.001531 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:53.999388 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 11:56:54.005734 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.005713 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 11:56:54.005979 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.005951 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gb6qt\"" Nov 02 11:56:54.006076 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.005990 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 11:56:54.006076 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.006031 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 11:56:54.006166 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.006113 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 11:56:54.020777 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.020753 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:54.048122 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.048023 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8hlf9gs6b7ohg\"" Nov 02 11:56:54.048122 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.048032 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 11:56:54.048122 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.048057 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 11:56:54.048122 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.048091 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 11:56:54.048338 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.048127 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 11:56:54.089722 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.089696 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.089826 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.089726 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.089826 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.089752 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.089976 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.089946 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090051 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090000 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090051 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090030 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090163 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090059 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090163 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090092 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090163 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090116 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090163 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090144 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090361 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090171 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090361 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090211 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090361 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090235 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4gz6\" (UniqueName: \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-kube-api-access-t4gz6\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090361 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090269 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090361 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090314 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090361 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090335 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090684 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090371 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090684 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090400 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090684 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090439 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090684 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090464 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090684 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090594 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.090926 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.090706 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.093130 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.092124 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.093130 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.092635 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.093130 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.093052 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.095079 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.094525 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.096040 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.096018 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.096892 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.096870 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.097853 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.097813 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 11:56:54.097945 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.097889 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 11:56:54.097945 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.097918 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.098138 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.098036 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.098138 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.098092 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.098557 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.098407 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.098557 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.098410 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.098557 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.098518 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.098738 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.098593 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.100195 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.100176 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.102664 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.102641 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.105208 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.105174 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.105484 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.105462 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 11:56:54.106851 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.106828 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 11:56:54.113258 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.113236 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.151475 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.151445 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4gz6\" (UniqueName: \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-kube-api-access-t4gz6\") pod \"prometheus-k8s-0\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.271044 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.270969 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:54.364612 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.364584 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:56:54.878555 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:54.878524 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 11:56:54.881327 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:54.881301 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1a37eb_bd9a_44e9_8de0_d1790513dc97.slice/crio-9d7e48361050761d9b2d3d76e3ae37fae3a38515e76201435d0a7fbd3d7b3a75 WatchSource:0}: Error finding container 9d7e48361050761d9b2d3d76e3ae37fae3a38515e76201435d0a7fbd3d7b3a75: Status 404 returned error can't find the container with id 9d7e48361050761d9b2d3d76e3ae37fae3a38515e76201435d0a7fbd3d7b3a75 Nov 02 11:56:55.267459 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:55.267420 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerStarted","Data":"9d7e48361050761d9b2d3d76e3ae37fae3a38515e76201435d0a7fbd3d7b3a75"} Nov 02 11:56:55.927352 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:55.927114 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c7779f96f-rpfvd"] Nov 02 11:56:55.956661 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:55.956635 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c8cbf96cd-j2zff"] Nov 02 11:56:55.977514 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:55.975716 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.043059 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.042811 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c8cbf96cd-j2zff"] Nov 02 11:56:56.109395 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.109360 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-serving-cert\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.109578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.109423 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-oauth-serving-cert\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.109578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.109529 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-tmp\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.109578 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.109574 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-trusted-ca-bundle\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.109727 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.109656 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-oauth-config\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.109787 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.109742 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-service-ca\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.109849 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.109786 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbr46\" (UniqueName: \"kubernetes.io/projected/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-kube-api-access-qbr46\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.109849 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.109836 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-config\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.210920 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.210848 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-service-ca\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.210920 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.210892 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbr46\" (UniqueName: \"kubernetes.io/projected/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-kube-api-access-qbr46\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.211169 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.210929 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-config\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.211169 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.210959 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-serving-cert\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.211169 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.211006 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-oauth-serving-cert\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.211169 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.211047 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-tmp\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.211169 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.211096 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-trusted-ca-bundle\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.211169 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.211136 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-oauth-config\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.211929 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.211706 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-tmp\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.212165 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.212115 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-trusted-ca-bundle\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.212741 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.212716 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-config\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.212873 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.212716 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-oauth-serving-cert\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.213180 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.213137 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-service-ca\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.214991 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.214972 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-oauth-config\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.215902 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.215859 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-serving-cert\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.273197 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.273173 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbr46\" (UniqueName: \"kubernetes.io/projected/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-kube-api-access-qbr46\") pod \"console-7c8cbf96cd-j2zff\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.308666 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.308642 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:56:56.705253 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.705216 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c8cbf96cd-j2zff"] Nov 02 11:56:56.709180 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:56:56.709150 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ed6225_0e61_41a1_88d7_6a8ae4dc357c.slice/crio-7a79f3e4988d198f08c8da7ea84f062dc438c42c144bc8ed4512918446edcc80 WatchSource:0}: Error finding container 7a79f3e4988d198f08c8da7ea84f062dc438c42c144bc8ed4512918446edcc80: Status 404 returned error can't find the container with id 7a79f3e4988d198f08c8da7ea84f062dc438c42c144bc8ed4512918446edcc80 Nov 02 11:56:56.990967 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:56.990891 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:56:57.115576 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:57.115393 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s2fqn" Nov 02 11:56:57.277604 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:57.277537 4312 generic.go:356] "Generic (PLEG): container finished" podID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerID="14ff9788519ead98666c909cec876c3e96de30494b790dfbe140edcb59908f43" exitCode=0 Nov 02 11:56:57.278029 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:57.277629 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerDied","Data":"14ff9788519ead98666c909cec876c3e96de30494b790dfbe140edcb59908f43"} Nov 02 11:56:57.282260 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:57.282236 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c8cbf96cd-j2zff" event={"ID":"54ed6225-0e61-41a1-88d7-6a8ae4dc357c","Type":"ContainerStarted","Data":"a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017"} Nov 02 11:56:57.282376 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:57.282282 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c8cbf96cd-j2zff" event={"ID":"54ed6225-0e61-41a1-88d7-6a8ae4dc357c","Type":"ContainerStarted","Data":"7a79f3e4988d198f08c8da7ea84f062dc438c42c144bc8ed4512918446edcc80"} Nov 02 11:56:57.425404 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:57.425370 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:57.533659 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:57.533435 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c8cbf96cd-j2zff" podStartSLOduration=2.533417039 podStartE2EDuration="2.533417039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:56:57.532538839 +0000 UTC m=+34.562775091" watchObservedRunningTime="2026-11-02 11:56:57.533417039 +0000 UTC m=+34.563653292" Nov 02 11:56:57.677055 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:56:57.677010 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:56:59.381057 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:56:59.380975 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 11:57:02.286736 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:02.286704 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:02.304360 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:02.304330 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerStarted","Data":"10dd98b0413d5ac93f6e1371e1ec52f799d96a1fdc41b0fc40a0129fd6b7ada5"} Nov 02 11:57:02.304522 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:02.304408 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerStarted","Data":"857ec75a0f935f433dca1fd7662517e0da4f2fcb5597d065d8763c228c09369e"} Nov 02 11:57:02.530930 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:02.530900 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:03.519809 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:03.519775 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:03.795864 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:03.795754 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:04.464296 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:04.464270 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:04.721085 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:04.721049 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:05.321197 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:05.321162 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerStarted","Data":"f11728f7394d68fe3155a0bb594ceaf3776c4e8f60ff3fcd8d3b406a93ce3254"} Nov 02 11:57:05.321375 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:05.321236 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerStarted","Data":"e32d1b82ab132a4c23101350b685b965d17b55ab9faae01c5034499849da73b8"} Nov 02 11:57:05.321375 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:05.321277 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerStarted","Data":"ed559ee9ca13325a3300cf92efa38f83202545cfa11d68e2d3527ce727d731f1"} Nov 02 11:57:05.321375 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:05.321314 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerStarted","Data":"d0fff4b82ec66da5f269a9d523031dc8a7c82c04f720d6893f42cf523ba8a7a1"} Nov 02 11:57:05.552326 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:05.552189 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.5735501689999998 podStartE2EDuration="12.552171749s" totalImagesPullingTime="8.958132442s" totalInitContainerRuntime="20.489138ms" isStatefulPod=false podCreationTimestamp="2026-11-02 11:56:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:57:05.507193373 +0000 UTC m=+42.537429636" watchObservedRunningTime="2026-11-02 11:57:05.552171749 +0000 UTC m=+42.582408001" Nov 02 11:57:05.958887 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:05.958834 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:06.210539 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:06.210472 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:06.309123 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:06.309099 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:57:06.309239 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:06.309179 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:57:06.314029 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:06.314011 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:57:06.328547 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:06.328529 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:57:06.584479 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:06.584407 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64796bd696-rnrjq"] Nov 02 11:57:09.365626 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:09.365598 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:15.438340 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:15.438303 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:15.690258 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:15.690175 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:19.977153 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:19.977115 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:20.026483 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:20.026455 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:20.390929 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:20.390864 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:20.953002 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:20.952944 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c7779f96f-rpfvd" podUID="8ccaf9ca-fa6b-44de-bafe-4705f66874ed" containerName="console" containerID="cri-o://3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c" gracePeriod=15 Nov 02 11:57:21.228603 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.228545 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:57:21.319792 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.319769 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-config\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-config\") pod \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " Nov 02 11:57:21.319905 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.319834 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-oauth-config\") pod \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " Nov 02 11:57:21.320086 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.320064 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-config" pod "8ccaf9ca-fa6b-44de-bafe-4705f66874ed" (UID: "8ccaf9ca-fa6b-44de-bafe-4705f66874ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:21.321917 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.321899 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-oauth-config" pod "8ccaf9ca-fa6b-44de-bafe-4705f66874ed" (UID: "8ccaf9ca-fa6b-44de-bafe-4705f66874ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:21.378637 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.378611 4312 generic.go:356] "Generic (PLEG): container finished" podID="8ccaf9ca-fa6b-44de-bafe-4705f66874ed" containerID="3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c" exitCode=0 Nov 02 11:57:21.378765 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.378669 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c7779f96f-rpfvd" Nov 02 11:57:21.378765 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.378709 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7779f96f-rpfvd" event={"ID":"8ccaf9ca-fa6b-44de-bafe-4705f66874ed","Type":"ContainerDied","Data":"3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c"} Nov 02 11:57:21.378765 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.378762 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c7779f96f-rpfvd" event={"ID":"8ccaf9ca-fa6b-44de-bafe-4705f66874ed","Type":"ContainerDied","Data":"bb39f5292ccd1f3895882ef14d224f5fd80f2bbd7692ee53f4c0ebd017fafa21"} Nov 02 11:57:21.378922 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.378792 4312 scope.go:122] "RemoveContainer" containerID="3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c" Nov 02 11:57:21.386522 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.386491 4312 scope.go:122] "RemoveContainer" containerID="3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c" Nov 02 11:57:21.386808 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:57:21.386792 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c\": container with ID starting with 3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c not found: ID does not exist" containerID="3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c" Nov 02 11:57:21.386855 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.386815 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c"} err="failed to get container status \"3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c\": rpc error: code = NotFound desc = could not find container \"3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c\": container with ID starting with 3960f49de9225656006a0f83f565b734de596b2417c83b47936de3a106599f5c not found: ID does not exist" Nov 02 11:57:21.421088 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421062 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-serving-cert\") pod \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " Nov 02 11:57:21.421199 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421118 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-tmp\") pod \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " Nov 02 11:57:21.421199 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421163 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-trusted-ca-bundle\") pod \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " Nov 02 11:57:21.421199 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421190 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-kube-api-access-qzjx7\" (UniqueName: \"kubernetes.io/projected/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-kube-api-access-qzjx7\") pod \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " Nov 02 11:57:21.421364 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421242 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-oauth-serving-cert\") pod \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " Nov 02 11:57:21.421364 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421271 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-service-ca\") pod \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\" (UID: \"8ccaf9ca-fa6b-44de-bafe-4705f66874ed\") " Nov 02 11:57:21.421471 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421395 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-tmp" pod "8ccaf9ca-fa6b-44de-bafe-4705f66874ed" (UID: "8ccaf9ca-fa6b-44de-bafe-4705f66874ed"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 11:57:21.421559 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421487 4312 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-config\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:21.421559 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421525 4312 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-tmp\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:21.421559 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421535 4312 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-oauth-config\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:21.421710 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421590 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-trusted-ca-bundle" pod "8ccaf9ca-fa6b-44de-bafe-4705f66874ed" (UID: "8ccaf9ca-fa6b-44de-bafe-4705f66874ed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:21.421710 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421602 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-oauth-serving-cert" pod "8ccaf9ca-fa6b-44de-bafe-4705f66874ed" (UID: "8ccaf9ca-fa6b-44de-bafe-4705f66874ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:21.421710 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.421638 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-service-ca" pod "8ccaf9ca-fa6b-44de-bafe-4705f66874ed" (UID: "8ccaf9ca-fa6b-44de-bafe-4705f66874ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:21.423078 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.423057 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-serving-cert" pod "8ccaf9ca-fa6b-44de-bafe-4705f66874ed" (UID: "8ccaf9ca-fa6b-44de-bafe-4705f66874ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:21.423364 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.423344 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-kube-api-access-qzjx7" pod "8ccaf9ca-fa6b-44de-bafe-4705f66874ed" (UID: "8ccaf9ca-fa6b-44de-bafe-4705f66874ed"). InnerVolumeSpecName "kube-api-access-qzjx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 11:57:21.522434 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.522371 4312 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-console-serving-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:21.522434 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.522396 4312 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-trusted-ca-bundle\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:21.522434 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.522409 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzjx7\" (UniqueName: \"kubernetes.io/projected/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-kube-api-access-qzjx7\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:21.522434 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.522426 4312 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-oauth-serving-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:21.522636 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.522439 4312 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ccaf9ca-fa6b-44de-bafe-4705f66874ed-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:21.797316 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.797261 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c7779f96f-rpfvd"] Nov 02 11:57:21.846600 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:21.846580 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c7779f96f-rpfvd"] Nov 02 11:57:23.073236 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:23.073205 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ccaf9ca-fa6b-44de-bafe-4705f66874ed" path="/var/lib/kubelet/pods/8ccaf9ca-fa6b-44de-bafe-4705f66874ed/volumes" Nov 02 11:57:24.322635 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:24.322603 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:24.569801 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:24.569772 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:29.371810 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:57:29.371771 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 11:57:31.603307 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.603253 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-64796bd696-rnrjq" podUID="2c8a1f9b-6089-4841-a311-dd0df9efe691" containerName="console" containerID="cri-o://592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881" gracePeriod=15 Nov 02 11:57:31.899008 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.898984 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:57:31.994504 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.994474 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c8a1f9b-6089-4841-a311-dd0df9efe691-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c8a1f9b-6089-4841-a311-dd0df9efe691-tmp\") pod \"2c8a1f9b-6089-4841-a311-dd0df9efe691\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " Nov 02 11:57:31.994635 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.994571 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-oauth-serving-cert\") pod \"2c8a1f9b-6089-4841-a311-dd0df9efe691\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " Nov 02 11:57:31.994635 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.994607 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-service-ca\") pod \"2c8a1f9b-6089-4841-a311-dd0df9efe691\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " Nov 02 11:57:31.994747 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.994652 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8a1f9b-6089-4841-a311-dd0df9efe691-kube-api-access-f2mwk\" (UniqueName: \"kubernetes.io/projected/2c8a1f9b-6089-4841-a311-dd0df9efe691-kube-api-access-f2mwk\") pod \"2c8a1f9b-6089-4841-a311-dd0df9efe691\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " Nov 02 11:57:31.994747 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.994686 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-config\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-config\") pod \"2c8a1f9b-6089-4841-a311-dd0df9efe691\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " Nov 02 11:57:31.994747 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.994729 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-serving-cert\") pod \"2c8a1f9b-6089-4841-a311-dd0df9efe691\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " Nov 02 11:57:31.994897 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.994759 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8a1f9b-6089-4841-a311-dd0df9efe691-tmp" pod "2c8a1f9b-6089-4841-a311-dd0df9efe691" (UID: "2c8a1f9b-6089-4841-a311-dd0df9efe691"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 11:57:31.994897 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.994768 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-oauth-config\") pod \"2c8a1f9b-6089-4841-a311-dd0df9efe691\" (UID: \"2c8a1f9b-6089-4841-a311-dd0df9efe691\") " Nov 02 11:57:31.995017 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.994982 4312 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c8a1f9b-6089-4841-a311-dd0df9efe691-tmp\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:31.995078 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.995049 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-service-ca" pod "2c8a1f9b-6089-4841-a311-dd0df9efe691" (UID: "2c8a1f9b-6089-4841-a311-dd0df9efe691"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:31.995078 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.995058 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-oauth-serving-cert" pod "2c8a1f9b-6089-4841-a311-dd0df9efe691" (UID: "2c8a1f9b-6089-4841-a311-dd0df9efe691"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:31.995167 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.995139 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-config" pod "2c8a1f9b-6089-4841-a311-dd0df9efe691" (UID: "2c8a1f9b-6089-4841-a311-dd0df9efe691"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:31.996796 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.996774 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-serving-cert" pod "2c8a1f9b-6089-4841-a311-dd0df9efe691" (UID: "2c8a1f9b-6089-4841-a311-dd0df9efe691"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:31.996947 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.996928 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-oauth-config" pod "2c8a1f9b-6089-4841-a311-dd0df9efe691" (UID: "2c8a1f9b-6089-4841-a311-dd0df9efe691"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:31.997009 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:31.996944 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8a1f9b-6089-4841-a311-dd0df9efe691-kube-api-access-f2mwk" pod "2c8a1f9b-6089-4841-a311-dd0df9efe691" (UID: "2c8a1f9b-6089-4841-a311-dd0df9efe691"). InnerVolumeSpecName "kube-api-access-f2mwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 11:57:32.096057 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.096027 4312 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-oauth-serving-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:32.096057 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.096055 4312 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:32.096221 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.096069 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2mwk\" (UniqueName: \"kubernetes.io/projected/2c8a1f9b-6089-4841-a311-dd0df9efe691-kube-api-access-f2mwk\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:32.096221 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.096084 4312 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-config\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:32.096221 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.096097 4312 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-serving-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:32.096221 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.096108 4312 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2c8a1f9b-6089-4841-a311-dd0df9efe691-console-oauth-config\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:32.416132 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.416099 4312 generic.go:356] "Generic (PLEG): container finished" podID="2c8a1f9b-6089-4841-a311-dd0df9efe691" containerID="592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881" exitCode=0 Nov 02 11:57:32.416276 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.416143 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64796bd696-rnrjq" event={"ID":"2c8a1f9b-6089-4841-a311-dd0df9efe691","Type":"ContainerDied","Data":"592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881"} Nov 02 11:57:32.416276 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.416150 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64796bd696-rnrjq" Nov 02 11:57:32.416276 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.416185 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64796bd696-rnrjq" event={"ID":"2c8a1f9b-6089-4841-a311-dd0df9efe691","Type":"ContainerDied","Data":"94910d01d0d579fe026bc52717a2e58c587b3f8ab949ccfcd5fc30369cd29c1c"} Nov 02 11:57:32.416276 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.416207 4312 scope.go:122] "RemoveContainer" containerID="592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881" Nov 02 11:57:32.425621 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.425605 4312 scope.go:122] "RemoveContainer" containerID="592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881" Nov 02 11:57:32.425914 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:57:32.425894 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881\": container with ID starting with 592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881 not found: ID does not exist" containerID="592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881" Nov 02 11:57:32.425959 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.425923 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881"} err="failed to get container status \"592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881\": rpc error: code = NotFound desc = could not find container \"592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881\": container with ID starting with 592e785e02b1e666e9fd0c17f2844242862d1fad2985205efc215ad3373bb881 not found: ID does not exist" Nov 02 11:57:32.578319 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.578291 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64796bd696-rnrjq"] Nov 02 11:57:32.627711 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:32.627687 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64796bd696-rnrjq"] Nov 02 11:57:33.076159 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:33.076130 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8a1f9b-6089-4841-a311-dd0df9efe691" path="/var/lib/kubelet/pods/2c8a1f9b-6089-4841-a311-dd0df9efe691/volumes" Nov 02 11:57:35.672890 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:35.672856 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:35.942675 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:35.942602 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:36.190730 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:36.190696 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:36.477112 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:36.477083 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:36.775577 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:36.775486 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:44.266780 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.266741 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 11:57:44.267270 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.267055 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="prometheus" containerID="cri-o://857ec75a0f935f433dca1fd7662517e0da4f2fcb5597d065d8763c228c09369e" gracePeriod=600 Nov 02 11:57:44.267270 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.267083 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy" containerID="cri-o://e32d1b82ab132a4c23101350b685b965d17b55ab9faae01c5034499849da73b8" gracePeriod=600 Nov 02 11:57:44.267270 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.267156 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f11728f7394d68fe3155a0bb594ceaf3776c4e8f60ff3fcd8d3b406a93ce3254" gracePeriod=600 Nov 02 11:57:44.267270 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.267114 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy-web" containerID="cri-o://ed559ee9ca13325a3300cf92efa38f83202545cfa11d68e2d3527ce727d731f1" gracePeriod=600 Nov 02 11:57:44.267270 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.267077 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="thanos-sidecar" containerID="cri-o://d0fff4b82ec66da5f269a9d523031dc8a7c82c04f720d6893f42cf523ba8a7a1" gracePeriod=600 Nov 02 11:57:44.267270 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.267253 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="config-reloader" containerID="cri-o://10dd98b0413d5ac93f6e1371e1ec52f799d96a1fdc41b0fc40a0129fd6b7ada5" gracePeriod=600 Nov 02 11:57:44.458064 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458035 4312 generic.go:356] "Generic (PLEG): container finished" podID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerID="f11728f7394d68fe3155a0bb594ceaf3776c4e8f60ff3fcd8d3b406a93ce3254" exitCode=0 Nov 02 11:57:44.458064 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458062 4312 generic.go:356] "Generic (PLEG): container finished" podID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerID="e32d1b82ab132a4c23101350b685b965d17b55ab9faae01c5034499849da73b8" exitCode=0 Nov 02 11:57:44.458064 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458071 4312 generic.go:356] "Generic (PLEG): container finished" podID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerID="ed559ee9ca13325a3300cf92efa38f83202545cfa11d68e2d3527ce727d731f1" exitCode=0 Nov 02 11:57:44.458269 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458077 4312 generic.go:356] "Generic (PLEG): container finished" podID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerID="d0fff4b82ec66da5f269a9d523031dc8a7c82c04f720d6893f42cf523ba8a7a1" exitCode=0 Nov 02 11:57:44.458269 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458082 4312 generic.go:356] "Generic (PLEG): container finished" podID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerID="10dd98b0413d5ac93f6e1371e1ec52f799d96a1fdc41b0fc40a0129fd6b7ada5" exitCode=0 Nov 02 11:57:44.458269 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458088 4312 generic.go:356] "Generic (PLEG): container finished" podID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerID="857ec75a0f935f433dca1fd7662517e0da4f2fcb5597d065d8763c228c09369e" exitCode=0 Nov 02 11:57:44.458269 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458161 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerDied","Data":"f11728f7394d68fe3155a0bb594ceaf3776c4e8f60ff3fcd8d3b406a93ce3254"} Nov 02 11:57:44.458269 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458212 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerDied","Data":"e32d1b82ab132a4c23101350b685b965d17b55ab9faae01c5034499849da73b8"} Nov 02 11:57:44.458269 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458237 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerDied","Data":"ed559ee9ca13325a3300cf92efa38f83202545cfa11d68e2d3527ce727d731f1"} Nov 02 11:57:44.458269 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458261 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerDied","Data":"d0fff4b82ec66da5f269a9d523031dc8a7c82c04f720d6893f42cf523ba8a7a1"} Nov 02 11:57:44.458477 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458283 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerDied","Data":"10dd98b0413d5ac93f6e1371e1ec52f799d96a1fdc41b0fc40a0129fd6b7ada5"} Nov 02 11:57:44.458477 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.458307 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerDied","Data":"857ec75a0f935f433dca1fd7662517e0da4f2fcb5597d065d8763c228c09369e"} Nov 02 11:57:44.509450 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.509426 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:44.585635 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.585579 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.585635 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.585611 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-tls\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.585635 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.585632 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-kubelet-serving-ca-bundle\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.585875 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.585651 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-trusted-ca-bundle\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.585875 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.585671 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-kube-api-access-t4gz6\" (UniqueName: \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-kube-api-access-t4gz6\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.585875 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.585697 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-0\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.585875 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.585727 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-metrics-client-certs\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.585875 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.585855 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-web-config\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-web-config\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586135 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.585895 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586135 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.585936 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-db\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586135 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.585971 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586135 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.586033 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-thanos-prometheus-http-client-file\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586135 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.586059 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-grpc-tls\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586135 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.586094 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-2\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586135 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.586102 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-trusted-ca-bundle" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:44.586135 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.586121 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-serving-certs-ca-bundle\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586574 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.586149 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config-out\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586574 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.586185 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-1\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586574 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.586216 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-kube-rbac-proxy\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586574 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.586243 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-metrics-client-ca\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586574 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.586298 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-tls-assets\" (UniqueName: \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-tls-assets\") pod \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\" (UID: \"fe1a37eb-bd9a-44e9-8de0-d1790513dc97\") " Nov 02 11:57:44.586574 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.586563 4312 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.587469 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.587209 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-serving-certs-ca-bundle" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:44.587824 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.587729 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-2" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:44.588399 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.587918 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-1" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:44.588399 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.588171 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-kubelet-serving-ca-bundle" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:44.588399 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.588289 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-kube-rbac-proxy-web" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:44.588399 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.588339 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-db" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 11:57:44.588399 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.588350 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-thanos-sidecar-tls" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:44.588399 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.588374 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-0" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:44.589218 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.588895 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-tls-assets" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 11:57:44.589218 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.589072 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:44.589331 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.589261 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-tls" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:44.589331 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.589280 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-metrics-client-ca" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:57:44.589943 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.589923 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-grpc-tls" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:44.590738 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.590714 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config-out" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 11:57:44.590946 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.590928 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-metrics-client-certs" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:44.591060 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.591039 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-kube-rbac-proxy" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:44.591158 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.591142 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-thanos-prometheus-http-client-file" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:44.591294 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.591277 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-kube-api-access-t4gz6" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "kube-api-access-t4gz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 11:57:44.599783 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.599764 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-web-config" pod "fe1a37eb-bd9a-44e9-8de0-d1790513dc97" (UID: "fe1a37eb-bd9a-44e9-8de0-d1790513dc97"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:57:44.687820 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687794 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4gz6\" (UniqueName: \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-kube-api-access-t4gz6\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.687820 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687821 4312 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687836 4312 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-metrics-client-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687851 4312 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-web-config\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687862 4312 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687875 4312 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-db\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687887 4312 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687901 4312 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687917 4312 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-grpc-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687930 4312 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687944 4312 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687957 4312 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-config-out\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687970 4312 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687984 4312 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-kube-rbac-proxy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.687999 4312 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-metrics-client-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688012 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.688013 4312 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-tls-assets\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688422 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.688027 4312 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688422 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.688041 4312 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:44.688422 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:44.688056 4312 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1a37eb-bd9a-44e9-8de0-d1790513dc97-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:57:45.465904 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.465872 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe1a37eb-bd9a-44e9-8de0-d1790513dc97","Type":"ContainerDied","Data":"9d7e48361050761d9b2d3d76e3ae37fae3a38515e76201435d0a7fbd3d7b3a75"} Nov 02 11:57:45.466305 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.465932 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.466305 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.465958 4312 scope.go:122] "RemoveContainer" containerID="f11728f7394d68fe3155a0bb594ceaf3776c4e8f60ff3fcd8d3b406a93ce3254" Nov 02 11:57:45.473711 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.473693 4312 scope.go:122] "RemoveContainer" containerID="e32d1b82ab132a4c23101350b685b965d17b55ab9faae01c5034499849da73b8" Nov 02 11:57:45.480225 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.480177 4312 scope.go:122] "RemoveContainer" containerID="ed559ee9ca13325a3300cf92efa38f83202545cfa11d68e2d3527ce727d731f1" Nov 02 11:57:45.486341 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.486320 4312 scope.go:122] "RemoveContainer" containerID="d0fff4b82ec66da5f269a9d523031dc8a7c82c04f720d6893f42cf523ba8a7a1" Nov 02 11:57:45.492587 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.492570 4312 scope.go:122] "RemoveContainer" containerID="10dd98b0413d5ac93f6e1371e1ec52f799d96a1fdc41b0fc40a0129fd6b7ada5" Nov 02 11:57:45.498995 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.498979 4312 scope.go:122] "RemoveContainer" containerID="857ec75a0f935f433dca1fd7662517e0da4f2fcb5597d065d8763c228c09369e" Nov 02 11:57:45.505570 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.505555 4312 scope.go:122] "RemoveContainer" containerID="14ff9788519ead98666c909cec876c3e96de30494b790dfbe140edcb59908f43" Nov 02 11:57:45.685739 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.684074 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 11:57:45.732744 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.732691 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 11:57:45.733622 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733604 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 11:57:45.733850 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733839 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy-thanos" Nov 02 11:57:45.733884 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733851 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy-thanos" Nov 02 11:57:45.733884 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733871 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="init-config-reloader" Nov 02 11:57:45.733884 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733876 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="init-config-reloader" Nov 02 11:57:45.733884 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733884 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy-web" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733889 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy-web" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733898 4312 cpu_manager.go:414] "containerMap: removing container" podUID="8ccaf9ca-fa6b-44de-bafe-4705f66874ed" containerName="console" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733902 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ccaf9ca-fa6b-44de-bafe-4705f66874ed" containerName="console" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733909 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="config-reloader" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733914 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="config-reloader" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733928 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="prometheus" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733932 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="prometheus" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733938 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2c8a1f9b-6089-4841-a311-dd0df9efe691" containerName="console" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733942 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8a1f9b-6089-4841-a311-dd0df9efe691" containerName="console" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733952 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733957 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733962 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="thanos-sidecar" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.733966 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="thanos-sidecar" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.734007 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy-web" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.734014 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="thanos-sidecar" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.734021 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ccaf9ca-fa6b-44de-bafe-4705f66874ed" containerName="console" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.734025 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="config-reloader" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.734030 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.734038 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="prometheus" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.734044 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" containerName="kube-rbac-proxy-thanos" Nov 02 11:57:45.734045 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.734049 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8a1f9b-6089-4841-a311-dd0df9efe691" containerName="console" Nov 02 11:57:45.737194 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.737179 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.789786 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.789756 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 11:57:45.789883 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.789812 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 11:57:45.790018 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790000 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 11:57:45.790091 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790076 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 11:57:45.790146 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790110 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 11:57:45.790146 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790089 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 11:57:45.790255 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790152 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 11:57:45.790367 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790343 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 11:57:45.790421 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790382 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 11:57:45.790460 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790442 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gb6qt\"" Nov 02 11:57:45.790520 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790453 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8hlf9gs6b7ohg\"" Nov 02 11:57:45.790520 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790485 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 11:57:45.790635 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790624 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 11:57:45.790697 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790624 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 11:57:45.790697 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.790677 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 11:57:45.795170 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795153 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795217 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795180 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795217 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795196 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795292 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795215 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-config\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795292 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795255 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795383 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795298 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795451 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795432 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795528 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795468 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb124f89-493d-435b-b51c-a78351f17aa1-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795528 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795516 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795660 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795552 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795660 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795579 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795660 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795628 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795660 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795652 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795786 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795690 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlskb\" (UniqueName: \"kubernetes.io/projected/fb124f89-493d-435b-b51c-a78351f17aa1-kube-api-access-qlskb\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795786 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795716 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795786 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795738 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795786 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795779 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb124f89-493d-435b-b51c-a78351f17aa1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795911 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795797 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795911 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795812 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.795911 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.795847 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.840349 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.840324 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 11:57:45.840423 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.840358 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 11:57:45.896090 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896049 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb124f89-493d-435b-b51c-a78351f17aa1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896172 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896097 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896172 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896121 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896172 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896163 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896267 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896195 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896267 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896222 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896267 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896246 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896374 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896270 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-config\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896374 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896306 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896374 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896338 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896551 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896377 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896551 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896403 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb124f89-493d-435b-b51c-a78351f17aa1-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896551 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896425 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896551 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896449 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896551 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896477 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896551 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896532 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896859 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896564 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896859 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896591 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlskb\" (UniqueName: \"kubernetes.io/projected/fb124f89-493d-435b-b51c-a78351f17aa1-kube-api-access-qlskb\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896859 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896642 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896859 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896664 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.896859 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.896775 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.897191 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.897167 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.897597 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.897442 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.898771 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.897828 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.898771 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.898288 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.899164 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.899144 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb124f89-493d-435b-b51c-a78351f17aa1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.899328 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.899282 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.900736 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.899808 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.900835 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.900806 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.900906 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.900893 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.901176 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.901127 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb124f89-493d-435b-b51c-a78351f17aa1-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.901271 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.901230 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.901376 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.901349 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.901465 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.901446 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.901564 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.901444 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-config\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.901634 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.901618 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.902650 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.902633 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.902782 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.902768 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb124f89-493d-435b-b51c-a78351f17aa1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.903675 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.903653 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb124f89-493d-435b-b51c-a78351f17aa1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.957916 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.957894 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlskb\" (UniqueName: \"kubernetes.io/projected/fb124f89-493d-435b-b51c-a78351f17aa1-kube-api-access-qlskb\") pod \"prometheus-k8s-0\" (UID: \"fb124f89-493d-435b-b51c-a78351f17aa1\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:45.968722 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:45.968702 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 11:57:46.052714 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.052668 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:46.480807 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.480771 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 11:57:46.489955 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:57:46.489925 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb124f89_493d_435b_b51c_a78351f17aa1.slice/crio-dc03375da38e1b6ae9537e2e1d31f99c790bfc90381ec85caedfe262fbf89169 WatchSource:0}: Error finding container dc03375da38e1b6ae9537e2e1d31f99c790bfc90381ec85caedfe262fbf89169: Status 404 returned error can't find the container with id dc03375da38e1b6ae9537e2e1d31f99c790bfc90381ec85caedfe262fbf89169 Nov 02 11:57:46.599795 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.599765 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bbbd8b6f7-cmm5w"] Nov 02 11:57:46.601631 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.601616 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.663677 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.663655 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bbbd8b6f7-cmm5w"] Nov 02 11:57:46.703288 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.703262 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-oauth-serving-cert\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.703400 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.703300 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-service-ca\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.703400 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.703333 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-oauth-config\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.703400 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.703388 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f57229b-a48d-40ad-8b50-e58a5b75d827-tmp\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.703554 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.703469 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-trusted-ca-bundle\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.703554 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.703507 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-config\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.703632 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.703571 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvqq\" (UniqueName: \"kubernetes.io/projected/1f57229b-a48d-40ad-8b50-e58a5b75d827-kube-api-access-8lvqq\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.703632 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.703591 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-serving-cert\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.804097 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.804042 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-trusted-ca-bundle\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.804097 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.804071 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-config\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.804097 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.804091 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvqq\" (UniqueName: \"kubernetes.io/projected/1f57229b-a48d-40ad-8b50-e58a5b75d827-kube-api-access-8lvqq\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.804322 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.804108 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-serving-cert\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.804322 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.804137 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-oauth-serving-cert\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.804322 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.804174 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-service-ca\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.804322 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.804220 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-oauth-config\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.804322 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.804244 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f57229b-a48d-40ad-8b50-e58a5b75d827-tmp\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.804890 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.804866 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-config\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.804995 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.804978 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-oauth-serving-cert\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.805058 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.805012 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-service-ca\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.805116 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.805065 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f57229b-a48d-40ad-8b50-e58a5b75d827-tmp\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.805116 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.805066 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-trusted-ca-bundle\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.806686 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.806669 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-serving-cert\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.806801 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.806784 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-oauth-config\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.865020 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.865001 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvqq\" (UniqueName: \"kubernetes.io/projected/1f57229b-a48d-40ad-8b50-e58a5b75d827-kube-api-access-8lvqq\") pod \"console-5bbbd8b6f7-cmm5w\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.925380 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.925357 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:46.979780 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:46.979747 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:47.073259 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:47.073203 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe1a37eb-bd9a-44e9-8de0-d1790513dc97" path="/var/lib/kubelet/pods/fe1a37eb-bd9a-44e9-8de0-d1790513dc97/volumes" Nov 02 11:57:47.230456 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:47.230428 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:47.293598 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:47.293486 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bbbd8b6f7-cmm5w"] Nov 02 11:57:47.296744 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:57:47.296720 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f57229b_a48d_40ad_8b50_e58a5b75d827.slice/crio-81fd5b9d8837dd42b4682cbda05a0ac7c720a16f70d50d33f32e84542bacf2e7 WatchSource:0}: Error finding container 81fd5b9d8837dd42b4682cbda05a0ac7c720a16f70d50d33f32e84542bacf2e7: Status 404 returned error can't find the container with id 81fd5b9d8837dd42b4682cbda05a0ac7c720a16f70d50d33f32e84542bacf2e7 Nov 02 11:57:47.479144 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:47.479094 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:47.480274 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:47.480250 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bbbd8b6f7-cmm5w" event={"ID":"1f57229b-a48d-40ad-8b50-e58a5b75d827","Type":"ContainerStarted","Data":"4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2"} Nov 02 11:57:47.480396 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:47.480329 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bbbd8b6f7-cmm5w" event={"ID":"1f57229b-a48d-40ad-8b50-e58a5b75d827","Type":"ContainerStarted","Data":"81fd5b9d8837dd42b4682cbda05a0ac7c720a16f70d50d33f32e84542bacf2e7"} Nov 02 11:57:47.481746 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:47.481721 4312 generic.go:356] "Generic (PLEG): container finished" podID="fb124f89-493d-435b-b51c-a78351f17aa1" containerID="919ed5b002cdf7ff049937fe3443da170bedee1cf1acf5ab2c72d843021a67df" exitCode=0 Nov 02 11:57:47.482041 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:47.481761 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb124f89-493d-435b-b51c-a78351f17aa1","Type":"ContainerDied","Data":"919ed5b002cdf7ff049937fe3443da170bedee1cf1acf5ab2c72d843021a67df"} Nov 02 11:57:47.482041 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:47.481804 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb124f89-493d-435b-b51c-a78351f17aa1","Type":"ContainerStarted","Data":"dc03375da38e1b6ae9537e2e1d31f99c790bfc90381ec85caedfe262fbf89169"} Nov 02 11:57:47.731676 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:47.731620 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5bbbd8b6f7-cmm5w" podStartSLOduration=1.731601825 podStartE2EDuration="1.731601825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:57:47.599108364 +0000 UTC m=+84.629344613" watchObservedRunningTime="2026-11-02 11:57:47.731601825 +0000 UTC m=+84.761838076" Nov 02 11:57:48.489091 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:48.489059 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb124f89-493d-435b-b51c-a78351f17aa1","Type":"ContainerStarted","Data":"4afa567f0f8f77541a4c256591ab6dd95a265fa705dd97ba989f316a7b0c4820"} Nov 02 11:57:48.489527 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:48.489119 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb124f89-493d-435b-b51c-a78351f17aa1","Type":"ContainerStarted","Data":"4c7a19021c0210332d1d8e566eb3b4b19a9447c119e6a9f686dbf81b6b8ef04b"} Nov 02 11:57:48.489527 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:48.489144 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb124f89-493d-435b-b51c-a78351f17aa1","Type":"ContainerStarted","Data":"a062e696114708fe69184ac3fd466bce61bf786e829abd214930d3708fdf7220"} Nov 02 11:57:48.489527 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:48.489170 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb124f89-493d-435b-b51c-a78351f17aa1","Type":"ContainerStarted","Data":"d560a4555a0497ffff3837c42d95fe44195c827a04810ec61d89f2fb0b5c9e5e"} Nov 02 11:57:48.489527 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:48.489195 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb124f89-493d-435b-b51c-a78351f17aa1","Type":"ContainerStarted","Data":"18ac1ce91f7f9b2c13a1bbe36c45083913a2d9d6762f4b3b534a0f7a61babff7"} Nov 02 11:57:48.489527 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:48.489226 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb124f89-493d-435b-b51c-a78351f17aa1","Type":"ContainerStarted","Data":"547998235dd15f9a647fa48f958a03f481dbf246728f368028b44be888e5fd1c"} Nov 02 11:57:48.621310 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:48.621252 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.597824748 podStartE2EDuration="3.621234785s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.410037ms" isStatefulPod=false podCreationTimestamp="2026-11-02 11:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:57:48.620465961 +0000 UTC m=+85.650702237" watchObservedRunningTime="2026-11-02 11:57:48.621234785 +0000 UTC m=+85.651471035" Nov 02 11:57:51.004827 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:51.004782 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:57:51.053653 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:51.053619 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:57:56.926251 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:56.926218 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:56.926687 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:56.926272 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:56.931219 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:56.931196 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:57.525891 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:57.525865 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:57:57.694510 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:57:57.694471 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c8cbf96cd-j2zff"] Nov 02 11:57:59.371724 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:57:59.371689 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 11:58:03.864776 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:03.864744 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:58:17.361921 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:17.361888 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:58:22.713648 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:22.713606 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c8cbf96cd-j2zff" podUID="54ed6225-0e61-41a1-88d7-6a8ae4dc357c" containerName="console" containerID="cri-o://a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017" gracePeriod=15 Nov 02 11:58:22.960757 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:22.960735 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:58:23.080275 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.080242 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-serving-cert\") pod \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " Nov 02 11:58:23.080385 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.080297 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-oauth-serving-cert\") pod \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " Nov 02 11:58:23.080385 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.080331 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-oauth-config\") pod \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " Nov 02 11:58:23.080385 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.080370 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-kube-api-access-qbr46\" (UniqueName: \"kubernetes.io/projected/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-kube-api-access-qbr46\") pod \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " Nov 02 11:58:23.080563 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.080417 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-config\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-config\") pod \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " Nov 02 11:58:23.080563 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.080448 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-tmp\") pod \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " Nov 02 11:58:23.080563 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.080537 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-trusted-ca-bundle\") pod \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " Nov 02 11:58:23.080563 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.080554 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-service-ca\") pod \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\" (UID: \"54ed6225-0e61-41a1-88d7-6a8ae4dc357c\") " Nov 02 11:58:23.081090 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.080970 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-tmp" pod "54ed6225-0e61-41a1-88d7-6a8ae4dc357c" (UID: "54ed6225-0e61-41a1-88d7-6a8ae4dc357c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 11:58:23.081090 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.081011 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-service-ca" pod "54ed6225-0e61-41a1-88d7-6a8ae4dc357c" (UID: "54ed6225-0e61-41a1-88d7-6a8ae4dc357c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:58:23.081664 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.081258 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-oauth-serving-cert" pod "54ed6225-0e61-41a1-88d7-6a8ae4dc357c" (UID: "54ed6225-0e61-41a1-88d7-6a8ae4dc357c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:58:23.081664 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.081442 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-trusted-ca-bundle" pod "54ed6225-0e61-41a1-88d7-6a8ae4dc357c" (UID: "54ed6225-0e61-41a1-88d7-6a8ae4dc357c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:58:23.081664 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.081490 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-config" pod "54ed6225-0e61-41a1-88d7-6a8ae4dc357c" (UID: "54ed6225-0e61-41a1-88d7-6a8ae4dc357c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:58:23.082893 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.082855 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-kube-api-access-qbr46" pod "54ed6225-0e61-41a1-88d7-6a8ae4dc357c" (UID: "54ed6225-0e61-41a1-88d7-6a8ae4dc357c"). InnerVolumeSpecName "kube-api-access-qbr46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 11:58:23.083319 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.083296 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-oauth-config" pod "54ed6225-0e61-41a1-88d7-6a8ae4dc357c" (UID: "54ed6225-0e61-41a1-88d7-6a8ae4dc357c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:58:23.083455 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.083435 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-serving-cert" pod "54ed6225-0e61-41a1-88d7-6a8ae4dc357c" (UID: "54ed6225-0e61-41a1-88d7-6a8ae4dc357c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:58:23.181400 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.181382 4312 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-tmp\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:58:23.181400 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.181401 4312 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-trusted-ca-bundle\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:58:23.181555 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.181411 4312 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:58:23.181555 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.181421 4312 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-serving-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:58:23.181555 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.181430 4312 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-oauth-serving-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:58:23.181555 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.181438 4312 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-oauth-config\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:58:23.181555 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.181446 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbr46\" (UniqueName: \"kubernetes.io/projected/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-kube-api-access-qbr46\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:58:23.181555 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.181456 4312 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54ed6225-0e61-41a1-88d7-6a8ae4dc357c-console-config\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:58:23.647389 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.647352 4312 generic.go:356] "Generic (PLEG): container finished" podID="54ed6225-0e61-41a1-88d7-6a8ae4dc357c" containerID="a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017" exitCode=0 Nov 02 11:58:23.647554 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.647417 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c8cbf96cd-j2zff" Nov 02 11:58:23.647554 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.647433 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c8cbf96cd-j2zff" event={"ID":"54ed6225-0e61-41a1-88d7-6a8ae4dc357c","Type":"ContainerDied","Data":"a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017"} Nov 02 11:58:23.647554 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.647480 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c8cbf96cd-j2zff" event={"ID":"54ed6225-0e61-41a1-88d7-6a8ae4dc357c","Type":"ContainerDied","Data":"7a79f3e4988d198f08c8da7ea84f062dc438c42c144bc8ed4512918446edcc80"} Nov 02 11:58:23.647554 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.647514 4312 scope.go:122] "RemoveContainer" containerID="a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017" Nov 02 11:58:23.655550 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.655529 4312 scope.go:122] "RemoveContainer" containerID="a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017" Nov 02 11:58:23.655839 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:58:23.655814 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017\": container with ID starting with a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017 not found: ID does not exist" containerID="a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017" Nov 02 11:58:23.655934 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.655843 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017"} err="failed to get container status \"a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017\": rpc error: code = NotFound desc = could not find container \"a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017\": container with ID starting with a6871bc62b1b89b78881a5da59b567fc5808851f76ff2f929574828e37abf017 not found: ID does not exist" Nov 02 11:58:23.811270 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.811241 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c8cbf96cd-j2zff"] Nov 02 11:58:23.860713 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:23.860692 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c8cbf96cd-j2zff"] Nov 02 11:58:25.073756 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:25.073706 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ed6225-0e61-41a1-88d7-6a8ae4dc357c" path="/var/lib/kubelet/pods/54ed6225-0e61-41a1-88d7-6a8ae4dc357c/volumes" Nov 02 11:58:29.374511 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:58:29.374455 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 11:58:46.053715 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:46.053628 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:58:46.070933 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:46.070908 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:58:46.745854 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:46.745820 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 11:58:54.256844 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:54.256804 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:58:56.177868 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.177840 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cbc99b8c8-2974f"] Nov 02 11:58:56.178341 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.178167 4312 cpu_manager.go:414] "containerMap: removing container" podUID="54ed6225-0e61-41a1-88d7-6a8ae4dc357c" containerName="console" Nov 02 11:58:56.178341 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.178180 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ed6225-0e61-41a1-88d7-6a8ae4dc357c" containerName="console" Nov 02 11:58:56.178341 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.178238 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ed6225-0e61-41a1-88d7-6a8ae4dc357c" containerName="console" Nov 02 11:58:56.180096 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.180080 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.203290 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.203264 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-console-serving-cert\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.203414 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.203295 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-oauth-serving-cert\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.203414 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.203322 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-trusted-ca-bundle\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.203552 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.203426 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-tmp\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.203552 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.203451 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29q7r\" (UniqueName: \"kubernetes.io/projected/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-kube-api-access-29q7r\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.203552 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.203517 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-console-oauth-config\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.203648 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.203560 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-console-config\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.203648 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.203624 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-service-ca\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.282415 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.282376 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cbc99b8c8-2974f"] Nov 02 11:58:56.304144 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.304124 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-tmp\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.304262 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.304149 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29q7r\" (UniqueName: \"kubernetes.io/projected/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-kube-api-access-29q7r\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.304262 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.304178 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-console-oauth-config\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.304262 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.304195 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-console-config\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.304262 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.304237 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-service-ca\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.304486 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.304282 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-console-serving-cert\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.304486 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.304303 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-oauth-serving-cert\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.304486 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.304328 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-trusted-ca-bundle\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.304703 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.304573 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-tmp\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.305038 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.305015 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-console-config\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.305125 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.305068 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-oauth-serving-cert\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.305125 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.305073 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-service-ca\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.305241 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.305183 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-trusted-ca-bundle\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.306632 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.306613 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-console-serving-cert\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.306816 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.306798 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-console-oauth-config\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.364515 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.364485 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29q7r\" (UniqueName: \"kubernetes.io/projected/fab746f0-70e4-4870-8a1d-3cd6ea60ef00-kube-api-access-29q7r\") pod \"console-5cbc99b8c8-2974f\" (UID: \"fab746f0-70e4-4870-8a1d-3cd6ea60ef00\") " pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.489611 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.489559 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:58:56.551583 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.551553 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:58:56.798833 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.798764 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:58:56.857417 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:56.857388 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cbc99b8c8-2974f"] Nov 02 11:58:56.890167 ip-10-0-141-170 kubenswrapper[4312]: W1102 11:58:56.890142 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab746f0_70e4_4870_8a1d_3cd6ea60ef00.slice/crio-7db7a6826bcd62ab1d8451fa4ec98331e859cfeed554602036aa93ca017542c5 WatchSource:0}: Error finding container 7db7a6826bcd62ab1d8451fa4ec98331e859cfeed554602036aa93ca017542c5: Status 404 returned error can't find the container with id 7db7a6826bcd62ab1d8451fa4ec98331e859cfeed554602036aa93ca017542c5 Nov 02 11:58:57.068899 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:57.068833 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:58:57.767887 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:57.767849 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cbc99b8c8-2974f" event={"ID":"fab746f0-70e4-4870-8a1d-3cd6ea60ef00","Type":"ContainerStarted","Data":"49f75ca30d0a3b277e00316cd837b98086a461753824b119ba44d91a2bcfc9e2"} Nov 02 11:58:57.768339 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:57.767903 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cbc99b8c8-2974f" event={"ID":"fab746f0-70e4-4870-8a1d-3cd6ea60ef00","Type":"ContainerStarted","Data":"7db7a6826bcd62ab1d8451fa4ec98331e859cfeed554602036aa93ca017542c5"} Nov 02 11:58:57.885758 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:58:57.885703 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5cbc99b8c8-2974f" podStartSLOduration=1.8856876059999998 podStartE2EDuration="1.885687606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 11:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 11:58:57.884943895 +0000 UTC m=+154.915180145" watchObservedRunningTime="2026-11-02 11:58:57.885687606 +0000 UTC m=+154.915923856" Nov 02 11:58:59.373742 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:58:59.373705 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 11:59:04.206881 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:04.206840 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:59:04.456579 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:04.456527 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:59:06.490357 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:06.490318 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:59:06.490782 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:06.490408 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:59:06.495413 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:06.495383 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:59:06.804706 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:06.804638 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cbc99b8c8-2974f" Nov 02 11:59:06.951149 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:06.951124 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bbbd8b6f7-cmm5w"] Nov 02 11:59:07.260882 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:07.260850 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:59:07.506728 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:07.506694 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:59:20.959646 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:20.959602 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:59:21.206630 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:21.206598 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 11:59:29.371733 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:59:29.371691 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 11:59:31.974600 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:31.974547 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5bbbd8b6f7-cmm5w" podUID="1f57229b-a48d-40ad-8b50-e58a5b75d827" containerName="console" containerID="cri-o://4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2" gracePeriod=15 Nov 02 11:59:32.210547 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.210529 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:59:32.353333 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353279 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-oauth-config\") pod \"1f57229b-a48d-40ad-8b50-e58a5b75d827\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " Nov 02 11:59:32.353333 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353323 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-oauth-serving-cert\") pod \"1f57229b-a48d-40ad-8b50-e58a5b75d827\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " Nov 02 11:59:32.353463 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353373 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f57229b-a48d-40ad-8b50-e58a5b75d827-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f57229b-a48d-40ad-8b50-e58a5b75d827-tmp\") pod \"1f57229b-a48d-40ad-8b50-e58a5b75d827\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " Nov 02 11:59:32.353463 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353417 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-trusted-ca-bundle\") pod \"1f57229b-a48d-40ad-8b50-e58a5b75d827\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " Nov 02 11:59:32.353463 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353446 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f57229b-a48d-40ad-8b50-e58a5b75d827-kube-api-access-8lvqq\" (UniqueName: \"kubernetes.io/projected/1f57229b-a48d-40ad-8b50-e58a5b75d827-kube-api-access-8lvqq\") pod \"1f57229b-a48d-40ad-8b50-e58a5b75d827\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " Nov 02 11:59:32.353626 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353485 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-service-ca\") pod \"1f57229b-a48d-40ad-8b50-e58a5b75d827\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " Nov 02 11:59:32.353626 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353538 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-serving-cert\") pod \"1f57229b-a48d-40ad-8b50-e58a5b75d827\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " Nov 02 11:59:32.353626 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353571 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-config\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-config\") pod \"1f57229b-a48d-40ad-8b50-e58a5b75d827\" (UID: \"1f57229b-a48d-40ad-8b50-e58a5b75d827\") " Nov 02 11:59:32.353774 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353707 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f57229b-a48d-40ad-8b50-e58a5b75d827-tmp" pod "1f57229b-a48d-40ad-8b50-e58a5b75d827" (UID: "1f57229b-a48d-40ad-8b50-e58a5b75d827"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 11:59:32.353832 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353783 4312 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f57229b-a48d-40ad-8b50-e58a5b75d827-tmp\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:59:32.353832 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353805 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-oauth-serving-cert" pod "1f57229b-a48d-40ad-8b50-e58a5b75d827" (UID: "1f57229b-a48d-40ad-8b50-e58a5b75d827"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:59:32.353937 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.353878 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-trusted-ca-bundle" pod "1f57229b-a48d-40ad-8b50-e58a5b75d827" (UID: "1f57229b-a48d-40ad-8b50-e58a5b75d827"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:59:32.354038 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.354010 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-service-ca" pod "1f57229b-a48d-40ad-8b50-e58a5b75d827" (UID: "1f57229b-a48d-40ad-8b50-e58a5b75d827"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:59:32.354131 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.354087 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-config" pod "1f57229b-a48d-40ad-8b50-e58a5b75d827" (UID: "1f57229b-a48d-40ad-8b50-e58a5b75d827"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 11:59:32.355336 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.355318 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-oauth-config" pod "1f57229b-a48d-40ad-8b50-e58a5b75d827" (UID: "1f57229b-a48d-40ad-8b50-e58a5b75d827"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:59:32.355717 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.355692 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f57229b-a48d-40ad-8b50-e58a5b75d827-kube-api-access-8lvqq" pod "1f57229b-a48d-40ad-8b50-e58a5b75d827" (UID: "1f57229b-a48d-40ad-8b50-e58a5b75d827"). InnerVolumeSpecName "kube-api-access-8lvqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 11:59:32.355779 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.355761 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-serving-cert" pod "1f57229b-a48d-40ad-8b50-e58a5b75d827" (UID: "1f57229b-a48d-40ad-8b50-e58a5b75d827"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 11:59:32.454206 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.454186 4312 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-trusted-ca-bundle\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:59:32.454206 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.454206 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lvqq\" (UniqueName: \"kubernetes.io/projected/1f57229b-a48d-40ad-8b50-e58a5b75d827-kube-api-access-8lvqq\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:59:32.454375 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.454218 4312 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:59:32.454375 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.454228 4312 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-serving-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:59:32.454375 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.454237 4312 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-config\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:59:32.454375 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.454245 4312 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1f57229b-a48d-40ad-8b50-e58a5b75d827-console-oauth-config\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:59:32.454375 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.454254 4312 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1f57229b-a48d-40ad-8b50-e58a5b75d827-oauth-serving-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 11:59:32.897088 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.897056 4312 generic.go:356] "Generic (PLEG): container finished" podID="1f57229b-a48d-40ad-8b50-e58a5b75d827" containerID="4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2" exitCode=0 Nov 02 11:59:32.897219 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.897107 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bbbd8b6f7-cmm5w" Nov 02 11:59:32.897219 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.897137 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bbbd8b6f7-cmm5w" event={"ID":"1f57229b-a48d-40ad-8b50-e58a5b75d827","Type":"ContainerDied","Data":"4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2"} Nov 02 11:59:32.897219 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.897191 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bbbd8b6f7-cmm5w" event={"ID":"1f57229b-a48d-40ad-8b50-e58a5b75d827","Type":"ContainerDied","Data":"81fd5b9d8837dd42b4682cbda05a0ac7c720a16f70d50d33f32e84542bacf2e7"} Nov 02 11:59:32.897219 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.897213 4312 scope.go:122] "RemoveContainer" containerID="4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2" Nov 02 11:59:32.905337 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.905324 4312 scope.go:122] "RemoveContainer" containerID="4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2" Nov 02 11:59:32.905583 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:59:32.905568 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2\": container with ID starting with 4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2 not found: ID does not exist" containerID="4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2" Nov 02 11:59:32.905639 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:32.905591 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2"} err="failed to get container status \"4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2\": rpc error: code = NotFound desc = could not find container \"4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2\": container with ID starting with 4ee4d89a18d2b8fcc5d40ec58ef13e1fbed17070551e87006dd6dc909ca9cbf2 not found: ID does not exist" Nov 02 11:59:33.060853 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:33.060830 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bbbd8b6f7-cmm5w"] Nov 02 11:59:33.111811 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:33.111787 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bbbd8b6f7-cmm5w"] Nov 02 11:59:35.072698 ip-10-0-141-170 kubenswrapper[4312]: I1102 11:59:35.072649 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f57229b-a48d-40ad-8b50-e58a5b75d827" path="/var/lib/kubelet/pods/1f57229b-a48d-40ad-8b50-e58a5b75d827/volumes" Nov 02 11:59:59.371837 ip-10-0-141-170 kubenswrapper[4312]: E1102 11:59:59.371803 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:00:04.556220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:00:04.556180 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:00:29.374618 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:00:29.374578 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:00:32.651932 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:00:32.651898 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:00:32.903034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:00:32.902949 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:00:59.372874 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:00:59.372835 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:01:04.922772 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:04.922733 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:16.572371 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:16.572333 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:16.858885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:16.858817 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:18.373051 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:18.373016 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:18.631237 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:18.631162 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:19.888290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:19.888252 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:25.269987 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:25.269939 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:25.531462 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:25.531373 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:25.843902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:25.843819 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:26.093472 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:26.093445 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:26.343260 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:26.343228 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:29.372643 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:01:29.372612 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:01:29.945759 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:01:29.945728 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:01:59.372452 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:01:59.372368 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:02:01.868100 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:01.868070 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:02.136010 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.135949 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:02.588978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.588946 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:02.715312 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.715276 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh"] Nov 02 12:02:02.715612 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.715596 4312 cpu_manager.go:414] "containerMap: removing container" podUID="1f57229b-a48d-40ad-8b50-e58a5b75d827" containerName="console" Nov 02 12:02:02.715666 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.715616 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f57229b-a48d-40ad-8b50-e58a5b75d827" containerName="console" Nov 02 12:02:02.715701 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.715694 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f57229b-a48d-40ad-8b50-e58a5b75d827" containerName="console" Nov 02 12:02:02.718772 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.718753 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.772939 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.772915 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 12:02:02.773276 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.773256 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 12:02:02.773604 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.773586 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 12:02:02.773787 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.773770 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-qclbq\"" Nov 02 12:02:02.822720 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.822694 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 12:02:02.822720 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.822717 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 12:02:02.878190 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.878134 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e54dcbb4-2e91-4ab5-a507-4fa251795f31-manifests\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.878468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.878187 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e54dcbb4-2e91-4ab5-a507-4fa251795f31-webhook-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.878468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.878256 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e54dcbb4-2e91-4ab5-a507-4fa251795f31-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.878468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.878297 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr7wn\" (UniqueName: \"kubernetes.io/projected/e54dcbb4-2e91-4ab5-a507-4fa251795f31-kube-api-access-sr7wn\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.878468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.878388 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e54dcbb4-2e91-4ab5-a507-4fa251795f31-tmp\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.878468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.878408 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e54dcbb4-2e91-4ab5-a507-4fa251795f31-metrics-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.903100 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.903077 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh"] Nov 02 12:02:02.909539 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.909518 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:02.978714 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.978690 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e54dcbb4-2e91-4ab5-a507-4fa251795f31-webhook-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.978801 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.978718 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e54dcbb4-2e91-4ab5-a507-4fa251795f31-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.978801 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.978756 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr7wn\" (UniqueName: \"kubernetes.io/projected/e54dcbb4-2e91-4ab5-a507-4fa251795f31-kube-api-access-sr7wn\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.978885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.978802 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e54dcbb4-2e91-4ab5-a507-4fa251795f31-tmp\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.978885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.978827 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e54dcbb4-2e91-4ab5-a507-4fa251795f31-metrics-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.978885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.978865 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e54dcbb4-2e91-4ab5-a507-4fa251795f31-manifests\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.979293 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.979271 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e54dcbb4-2e91-4ab5-a507-4fa251795f31-manifests\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.981010 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.980985 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e54dcbb4-2e91-4ab5-a507-4fa251795f31-tmp\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.981330 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.981306 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e54dcbb4-2e91-4ab5-a507-4fa251795f31-metrics-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.981594 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.981568 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e54dcbb4-2e91-4ab5-a507-4fa251795f31-webhook-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:02.981736 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:02.981714 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e54dcbb4-2e91-4ab5-a507-4fa251795f31-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:03.043163 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:03.043136 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr7wn\" (UniqueName: \"kubernetes.io/projected/e54dcbb4-2e91-4ab5-a507-4fa251795f31-kube-api-access-sr7wn\") pod \"opendatahub-operator-controller-manager-9776d789f-fhjfh\" (UID: \"e54dcbb4-2e91-4ab5-a507-4fa251795f31\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:03.230559 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:03.230537 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:03.333569 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:03.333548 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:03.495079 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:03.495020 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:03.907969 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:03.907938 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh"] Nov 02 12:02:03.909353 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:02:03.909329 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54dcbb4_2e91_4ab5_a507_4fa251795f31.slice/crio-a26029db218161174caac940b8667782c707fc7b30a13ef99991f27b8bb051ff WatchSource:0}: Error finding container a26029db218161174caac940b8667782c707fc7b30a13ef99991f27b8bb051ff: Status 404 returned error can't find the container with id a26029db218161174caac940b8667782c707fc7b30a13ef99991f27b8bb051ff Nov 02 12:02:03.910955 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:03.910931 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 12:02:04.417402 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:04.417369 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" event={"ID":"e54dcbb4-2e91-4ab5-a507-4fa251795f31","Type":"ContainerStarted","Data":"a26029db218161174caac940b8667782c707fc7b30a13ef99991f27b8bb051ff"} Nov 02 12:02:05.262002 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.261959 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:05.389708 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.389679 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv"] Nov 02 12:02:05.393459 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.392776 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv" Nov 02 12:02:05.447693 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.447670 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 12:02:05.497587 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.497552 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghrzs\" (UniqueName: \"kubernetes.io/projected/16e242ee-c63d-46e7-b866-dc3b36a0ac72-kube-api-access-ghrzs\") pod \"jobset-operator-674cd59dc5-bdzcv\" (UID: \"16e242ee-c63d-46e7-b866-dc3b36a0ac72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv" Nov 02 12:02:05.497717 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.497598 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e242ee-c63d-46e7-b866-dc3b36a0ac72-tmp\") pod \"jobset-operator-674cd59dc5-bdzcv\" (UID: \"16e242ee-c63d-46e7-b866-dc3b36a0ac72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv" Nov 02 12:02:05.497810 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.497797 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 12:02:05.497947 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.497936 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-d5njb\"" Nov 02 12:02:05.498953 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.498892 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv"] Nov 02 12:02:05.509508 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.509472 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:05.598759 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.598698 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghrzs\" (UniqueName: \"kubernetes.io/projected/16e242ee-c63d-46e7-b866-dc3b36a0ac72-kube-api-access-ghrzs\") pod \"jobset-operator-674cd59dc5-bdzcv\" (UID: \"16e242ee-c63d-46e7-b866-dc3b36a0ac72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv" Nov 02 12:02:05.598889 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.598759 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e242ee-c63d-46e7-b866-dc3b36a0ac72-tmp\") pod \"jobset-operator-674cd59dc5-bdzcv\" (UID: \"16e242ee-c63d-46e7-b866-dc3b36a0ac72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv" Nov 02 12:02:05.599177 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.599157 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e242ee-c63d-46e7-b866-dc3b36a0ac72-tmp\") pod \"jobset-operator-674cd59dc5-bdzcv\" (UID: \"16e242ee-c63d-46e7-b866-dc3b36a0ac72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv" Nov 02 12:02:05.659811 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.659756 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghrzs\" (UniqueName: \"kubernetes.io/projected/16e242ee-c63d-46e7-b866-dc3b36a0ac72-kube-api-access-ghrzs\") pod \"jobset-operator-674cd59dc5-bdzcv\" (UID: \"16e242ee-c63d-46e7-b866-dc3b36a0ac72\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv" Nov 02 12:02:05.721103 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.721078 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv" Nov 02 12:02:05.889934 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:05.889856 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:06.123536 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:06.123483 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv"] Nov 02 12:02:06.127745 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:02:06.127705 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16e242ee_c63d_46e7_b866_dc3b36a0ac72.slice/crio-14e649590fbe2032125cbe0970827de9134488712dac9296197d83867da3fa7f WatchSource:0}: Error finding container 14e649590fbe2032125cbe0970827de9134488712dac9296197d83867da3fa7f: Status 404 returned error can't find the container with id 14e649590fbe2032125cbe0970827de9134488712dac9296197d83867da3fa7f Nov 02 12:02:06.261402 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:06.261352 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:06.427145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:06.427112 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv" event={"ID":"16e242ee-c63d-46e7-b866-dc3b36a0ac72","Type":"ContainerStarted","Data":"14e649590fbe2032125cbe0970827de9134488712dac9296197d83867da3fa7f"} Nov 02 12:02:06.525653 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:06.525579 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:06.773648 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:06.773615 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:07.432167 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:07.432139 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" event={"ID":"e54dcbb4-2e91-4ab5-a507-4fa251795f31","Type":"ContainerStarted","Data":"b2e3c3d52100bc03edabd766f5fcf436f678cb84e8750e3779ef21ae3a4c90a9"} Nov 02 12:02:08.439233 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:08.439200 4312 generic.go:356] "Generic (PLEG): container finished" podID="e54dcbb4-2e91-4ab5-a507-4fa251795f31" containerID="b2e3c3d52100bc03edabd766f5fcf436f678cb84e8750e3779ef21ae3a4c90a9" exitCode=0 Nov 02 12:02:08.439722 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:08.439279 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" event={"ID":"e54dcbb4-2e91-4ab5-a507-4fa251795f31","Type":"ContainerDied","Data":"b2e3c3d52100bc03edabd766f5fcf436f678cb84e8750e3779ef21ae3a4c90a9"} Nov 02 12:02:09.282019 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:09.281989 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:09.447277 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:09.447242 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" event={"ID":"e54dcbb4-2e91-4ab5-a507-4fa251795f31","Type":"ContainerStarted","Data":"787955cf191dfb445204b2971e20059fa78d91204cfe7abc86ce49d6b790c1dd"} Nov 02 12:02:09.447759 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:09.447406 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:09.645458 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:09.645348 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" podStartSLOduration=3.118460617 podStartE2EDuration="7.645327006s" totalImagesPullingTime="4.066903385s" totalInitContainerRuntime="459.963004ms" isStatefulPod=false podCreationTimestamp="2026-11-02 12:02:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:02:09.639339187 +0000 UTC m=+346.669575437" watchObservedRunningTime="2026-11-02 12:02:09.645327006 +0000 UTC m=+346.675563257" Nov 02 12:02:10.451943 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:10.451902 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv" event={"ID":"16e242ee-c63d-46e7-b866-dc3b36a0ac72","Type":"ContainerStarted","Data":"ab1c913ea6319f4c8972f87e69dfd95c5747271fc9ae367edc421d3f9f4d4226"} Nov 02 12:02:10.569105 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:10.569006 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bdzcv" podStartSLOduration=1.933740708 podStartE2EDuration="5.568990082s" totalImagesPullingTime="3.635249374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:02:10.568027225 +0000 UTC m=+347.598263472" watchObservedRunningTime="2026-11-02 12:02:10.568990082 +0000 UTC m=+347.599226333" Nov 02 12:02:14.277534 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:14.277475 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:16.463791 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:16.463754 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:17.873676 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:17.873642 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:18.662040 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:18.662004 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:19.140426 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:19.140358 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:19.520195 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:19.520164 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:20.454527 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:20.454485 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-9776d789f-fhjfh" Nov 02 12:02:21.334941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:21.334903 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:21.755745 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:21.755703 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:23.584414 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:23.584372 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:23.923311 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:23.923280 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:25.379942 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:25.379902 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:25.946291 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:25.946253 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:26.877433 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:26.877400 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:27.178379 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:27.178345 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:28.504509 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.504465 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7"] Nov 02 12:02:28.514937 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.514914 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7" Nov 02 12:02:28.585291 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.585268 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 12:02:28.585401 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.585349 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-rz9bv\"" Nov 02 12:02:28.634850 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.634819 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 12:02:28.636134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.636112 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7"] Nov 02 12:02:28.670940 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.670918 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwzf4\" (UniqueName: \"kubernetes.io/projected/8443cbf2-bd01-4af6-b94d-9515b88acf8e-kube-api-access-xwzf4\") pod \"cert-manager-cainjector-5c57bbd4fc-pm8v7\" (UID: \"8443cbf2-bd01-4af6-b94d-9515b88acf8e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7" Nov 02 12:02:28.671038 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.670963 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8443cbf2-bd01-4af6-b94d-9515b88acf8e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-pm8v7\" (UID: \"8443cbf2-bd01-4af6-b94d-9515b88acf8e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7" Nov 02 12:02:28.698694 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.698669 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:28.771420 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.771365 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzf4\" (UniqueName: \"kubernetes.io/projected/8443cbf2-bd01-4af6-b94d-9515b88acf8e-kube-api-access-xwzf4\") pod \"cert-manager-cainjector-5c57bbd4fc-pm8v7\" (UID: \"8443cbf2-bd01-4af6-b94d-9515b88acf8e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7" Nov 02 12:02:28.771420 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.771392 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8443cbf2-bd01-4af6-b94d-9515b88acf8e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-pm8v7\" (UID: \"8443cbf2-bd01-4af6-b94d-9515b88acf8e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7" Nov 02 12:02:28.832428 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.832401 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8443cbf2-bd01-4af6-b94d-9515b88acf8e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-pm8v7\" (UID: \"8443cbf2-bd01-4af6-b94d-9515b88acf8e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7" Nov 02 12:02:28.832549 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:28.832533 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzf4\" (UniqueName: \"kubernetes.io/projected/8443cbf2-bd01-4af6-b94d-9515b88acf8e-kube-api-access-xwzf4\") pod \"cert-manager-cainjector-5c57bbd4fc-pm8v7\" (UID: \"8443cbf2-bd01-4af6-b94d-9515b88acf8e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7" Nov 02 12:02:29.126025 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.125960 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7" Nov 02 12:02:29.140412 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.140387 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:29.249502 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.249474 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt"] Nov 02 12:02:29.252072 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.252057 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt" Nov 02 12:02:29.315835 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.315815 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 12:02:29.315960 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.315947 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 12:02:29.316019 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.316005 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4qpc2\"" Nov 02 12:02:29.367054 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.367021 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt"] Nov 02 12:02:29.373482 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:02:29.373456 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:02:29.377953 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.377897 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sggqj\" (UniqueName: \"kubernetes.io/projected/39bf2d16-2a6d-4822-8d9c-b768de423729-kube-api-access-sggqj\") pod \"authorino-operator-7b7bb4bc77-wqxtt\" (UID: \"39bf2d16-2a6d-4822-8d9c-b768de423729\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt" Nov 02 12:02:29.450237 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.450209 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:29.478700 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.478676 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sggqj\" (UniqueName: \"kubernetes.io/projected/39bf2d16-2a6d-4822-8d9c-b768de423729-kube-api-access-sggqj\") pod \"authorino-operator-7b7bb4bc77-wqxtt\" (UID: \"39bf2d16-2a6d-4822-8d9c-b768de423729\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt" Nov 02 12:02:29.558509 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.558475 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7"] Nov 02 12:02:29.560890 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:02:29.560868 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8443cbf2_bd01_4af6_b94d_9515b88acf8e.slice/crio-ed447e2c3d52120292a13582bd7a63ea6ae35de6a09e5ad1b0c42a6277be2d39 WatchSource:0}: Error finding container ed447e2c3d52120292a13582bd7a63ea6ae35de6a09e5ad1b0c42a6277be2d39: Status 404 returned error can't find the container with id ed447e2c3d52120292a13582bd7a63ea6ae35de6a09e5ad1b0c42a6277be2d39 Nov 02 12:02:29.561263 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.561243 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sggqj\" (UniqueName: \"kubernetes.io/projected/39bf2d16-2a6d-4822-8d9c-b768de423729-kube-api-access-sggqj\") pod \"authorino-operator-7b7bb4bc77-wqxtt\" (UID: \"39bf2d16-2a6d-4822-8d9c-b768de423729\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt" Nov 02 12:02:29.563160 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.563139 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt" Nov 02 12:02:29.748787 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.748761 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:29.932992 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:29.932966 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt"] Nov 02 12:02:29.936373 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:02:29.936349 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bf2d16_2a6d_4822_8d9c_b768de423729.slice/crio-35728821f5e64d2f619e0f592cf40907ec452f8c931cfa66c99487ccd378d2b0 WatchSource:0}: Error finding container 35728821f5e64d2f619e0f592cf40907ec452f8c931cfa66c99487ccd378d2b0: Status 404 returned error can't find the container with id 35728821f5e64d2f619e0f592cf40907ec452f8c931cfa66c99487ccd378d2b0 Nov 02 12:02:30.526740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:30.526707 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt" event={"ID":"39bf2d16-2a6d-4822-8d9c-b768de423729","Type":"ContainerStarted","Data":"35728821f5e64d2f619e0f592cf40907ec452f8c931cfa66c99487ccd378d2b0"} Nov 02 12:02:30.527858 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:30.527837 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7" event={"ID":"8443cbf2-bd01-4af6-b94d-9515b88acf8e","Type":"ContainerStarted","Data":"ed447e2c3d52120292a13582bd7a63ea6ae35de6a09e5ad1b0c42a6277be2d39"} Nov 02 12:02:30.948193 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:30.948161 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:31.200694 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:31.200624 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:33.125688 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.125659 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh"] Nov 02 12:02:33.129220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.128863 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" Nov 02 12:02:33.174570 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.174533 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh"] Nov 02 12:02:33.181214 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.181190 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-7fdqn\"" Nov 02 12:02:33.215898 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.215873 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6425998-62cf-48d1-b38b-c78c91b81942-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sqhjh\" (UID: \"c6425998-62cf-48d1-b38b-c78c91b81942\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" Nov 02 12:02:33.216155 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.216128 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghn48\" (UniqueName: \"kubernetes.io/projected/c6425998-62cf-48d1-b38b-c78c91b81942-kube-api-access-ghn48\") pod \"cert-manager-webhook-5dbbbf75cc-sqhjh\" (UID: \"c6425998-62cf-48d1-b38b-c78c91b81942\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" Nov 02 12:02:33.282088 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.281954 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:33.317343 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.317311 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghn48\" (UniqueName: \"kubernetes.io/projected/c6425998-62cf-48d1-b38b-c78c91b81942-kube-api-access-ghn48\") pod \"cert-manager-webhook-5dbbbf75cc-sqhjh\" (UID: \"c6425998-62cf-48d1-b38b-c78c91b81942\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" Nov 02 12:02:33.317519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.317428 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6425998-62cf-48d1-b38b-c78c91b81942-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sqhjh\" (UID: \"c6425998-62cf-48d1-b38b-c78c91b81942\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" Nov 02 12:02:33.386246 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.386180 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c6425998-62cf-48d1-b38b-c78c91b81942-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sqhjh\" (UID: \"c6425998-62cf-48d1-b38b-c78c91b81942\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" Nov 02 12:02:33.399900 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.399878 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghn48\" (UniqueName: \"kubernetes.io/projected/c6425998-62cf-48d1-b38b-c78c91b81942-kube-api-access-ghn48\") pod \"cert-manager-webhook-5dbbbf75cc-sqhjh\" (UID: \"c6425998-62cf-48d1-b38b-c78c91b81942\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" Nov 02 12:02:33.441703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.441678 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" Nov 02 12:02:33.530039 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.530008 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:33.817917 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:33.817889 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh"] Nov 02 12:02:33.819741 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:02:33.819702 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6425998_62cf_48d1_b38b_c78c91b81942.slice/crio-b5d84403ac158989c73487fb6ef5c9bde5e0d2f4b905d48e588895b301507d72 WatchSource:0}: Error finding container b5d84403ac158989c73487fb6ef5c9bde5e0d2f4b905d48e588895b301507d72: Status 404 returned error can't find the container with id b5d84403ac158989c73487fb6ef5c9bde5e0d2f4b905d48e588895b301507d72 Nov 02 12:02:34.549119 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:34.549087 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt" event={"ID":"39bf2d16-2a6d-4822-8d9c-b768de423729","Type":"ContainerStarted","Data":"2473c465db60f52a426742c8666ff346713b9a23c7433e9a03cdfd11042977a3"} Nov 02 12:02:34.549669 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:34.549231 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt" Nov 02 12:02:34.550675 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:34.550647 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7" event={"ID":"8443cbf2-bd01-4af6-b94d-9515b88acf8e","Type":"ContainerStarted","Data":"3e54def2a3abe9eb6040075d1713918141124c2750641ba988484cb5d5f34cb8"} Nov 02 12:02:34.551780 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:34.551757 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" event={"ID":"c6425998-62cf-48d1-b38b-c78c91b81942","Type":"ContainerStarted","Data":"b5d84403ac158989c73487fb6ef5c9bde5e0d2f4b905d48e588895b301507d72"} Nov 02 12:02:34.666991 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:34.666951 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt" podStartSLOduration=1.902332263 podStartE2EDuration="5.666937801s" totalImagesPullingTime="3.764605538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:02:34.665595237 +0000 UTC m=+371.695831487" watchObservedRunningTime="2026-11-02 12:02:34.666937801 +0000 UTC m=+371.697174047" Nov 02 12:02:34.780491 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:34.780445 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-pm8v7" podStartSLOduration=2.1988193 podStartE2EDuration="6.780431691s" totalImagesPullingTime="4.581612391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:02:34.779297479 +0000 UTC m=+371.809533731" watchObservedRunningTime="2026-11-02 12:02:34.780431691 +0000 UTC m=+371.810667919" Nov 02 12:02:35.064688 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.064661 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr"] Nov 02 12:02:35.067699 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.067680 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" Nov 02 12:02:35.121802 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.121776 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 02 12:02:35.121935 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.121814 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-98q7s\"" Nov 02 12:02:35.121935 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.121905 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 02 12:02:35.129010 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.128986 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr"] Nov 02 12:02:35.135369 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.135344 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/89fbd2ad-bb31-409d-90ba-b7d358be49a7-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-ck9gr\" (UID: \"89fbd2ad-bb31-409d-90ba-b7d358be49a7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" Nov 02 12:02:35.135555 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.135528 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/89fbd2ad-bb31-409d-90ba-b7d358be49a7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-ck9gr\" (UID: \"89fbd2ad-bb31-409d-90ba-b7d358be49a7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" Nov 02 12:02:35.135701 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.135679 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx52x\" (UniqueName: \"kubernetes.io/projected/89fbd2ad-bb31-409d-90ba-b7d358be49a7-kube-api-access-qx52x\") pod \"kuadrant-console-plugin-66c4fb575d-ck9gr\" (UID: \"89fbd2ad-bb31-409d-90ba-b7d358be49a7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" Nov 02 12:02:35.159798 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.159771 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:35.236276 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.236251 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/89fbd2ad-bb31-409d-90ba-b7d358be49a7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-ck9gr\" (UID: \"89fbd2ad-bb31-409d-90ba-b7d358be49a7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" Nov 02 12:02:35.236376 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.236289 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx52x\" (UniqueName: \"kubernetes.io/projected/89fbd2ad-bb31-409d-90ba-b7d358be49a7-kube-api-access-qx52x\") pod \"kuadrant-console-plugin-66c4fb575d-ck9gr\" (UID: \"89fbd2ad-bb31-409d-90ba-b7d358be49a7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" Nov 02 12:02:35.236424 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.236390 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/89fbd2ad-bb31-409d-90ba-b7d358be49a7-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-ck9gr\" (UID: \"89fbd2ad-bb31-409d-90ba-b7d358be49a7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" Nov 02 12:02:35.236933 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.236914 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/89fbd2ad-bb31-409d-90ba-b7d358be49a7-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-ck9gr\" (UID: \"89fbd2ad-bb31-409d-90ba-b7d358be49a7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" Nov 02 12:02:35.238617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.238599 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/89fbd2ad-bb31-409d-90ba-b7d358be49a7-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-ck9gr\" (UID: \"89fbd2ad-bb31-409d-90ba-b7d358be49a7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" Nov 02 12:02:35.299681 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.299631 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx52x\" (UniqueName: \"kubernetes.io/projected/89fbd2ad-bb31-409d-90ba-b7d358be49a7-kube-api-access-qx52x\") pod \"kuadrant-console-plugin-66c4fb575d-ck9gr\" (UID: \"89fbd2ad-bb31-409d-90ba-b7d358be49a7\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" Nov 02 12:02:35.390243 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.390209 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" Nov 02 12:02:35.407383 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.407359 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:35.557350 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.557270 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" event={"ID":"c6425998-62cf-48d1-b38b-c78c91b81942","Type":"ContainerStarted","Data":"955e61537c4e52b281f6f4d1d3267cc541fdde2b422c0a3223a040ee1849afab"} Nov 02 12:02:35.557727 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.557587 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" Nov 02 12:02:35.676261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.676214 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" podStartSLOduration=1.456099788 podStartE2EDuration="2.676202426s" totalImagesPullingTime="1.220102638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:02:35.674890835 +0000 UTC m=+372.705127086" watchObservedRunningTime="2026-11-02 12:02:35.676202426 +0000 UTC m=+372.706438676" Nov 02 12:02:35.788611 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.788589 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr"] Nov 02 12:02:35.791254 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:02:35.791228 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89fbd2ad_bb31_409d_90ba_b7d358be49a7.slice/crio-2c685ceda3236bd7cea89ba202d4a579d3b816b294ea6e74c6fa1c9a36be3f7e WatchSource:0}: Error finding container 2c685ceda3236bd7cea89ba202d4a579d3b816b294ea6e74c6fa1c9a36be3f7e: Status 404 returned error can't find the container with id 2c685ceda3236bd7cea89ba202d4a579d3b816b294ea6e74c6fa1c9a36be3f7e Nov 02 12:02:35.791698 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:35.791674 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:36.124230 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:36.124203 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:36.562306 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:36.562274 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" event={"ID":"89fbd2ad-bb31-409d-90ba-b7d358be49a7","Type":"ContainerStarted","Data":"2c685ceda3236bd7cea89ba202d4a579d3b816b294ea6e74c6fa1c9a36be3f7e"} Nov 02 12:02:38.039312 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:38.039284 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:39.122387 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:39.122315 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:39.376626 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:39.376538 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:39.824041 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:39.824002 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:41.092961 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.092911 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:41.444254 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.444214 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:41.551623 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.551596 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl"] Nov 02 12:02:41.554932 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.554918 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" Nov 02 12:02:41.564260 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.564230 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sqhjh" Nov 02 12:02:41.584261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.584227 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" event={"ID":"89fbd2ad-bb31-409d-90ba-b7d358be49a7","Type":"ContainerStarted","Data":"fd4f03adf3458e01e091aa432d4cdfec25fb350fb38fb812b3f1fc8a54287c1e"} Nov 02 12:02:41.585608 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.585589 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01362f25-65e7-4ec6-afcc-aa1410815d17-tls-certs\") pod \"kube-auth-proxy-85ccdd577f-llvhl\" (UID: \"01362f25-65e7-4ec6-afcc-aa1410815d17\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" Nov 02 12:02:41.585696 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.585629 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01362f25-65e7-4ec6-afcc-aa1410815d17-tmp\") pod \"kube-auth-proxy-85ccdd577f-llvhl\" (UID: \"01362f25-65e7-4ec6-afcc-aa1410815d17\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" Nov 02 12:02:41.585771 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.585748 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qt4d\" (UniqueName: \"kubernetes.io/projected/01362f25-65e7-4ec6-afcc-aa1410815d17-kube-api-access-6qt4d\") pod \"kube-auth-proxy-85ccdd577f-llvhl\" (UID: \"01362f25-65e7-4ec6-afcc-aa1410815d17\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" Nov 02 12:02:41.624610 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.624547 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 12:02:41.624610 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.624568 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-lw8pv\"" Nov 02 12:02:41.624747 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.624636 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 12:02:41.631591 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.631577 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 12:02:41.662742 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.662723 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 12:02:41.664170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.664149 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl"] Nov 02 12:02:41.686204 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.686181 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01362f25-65e7-4ec6-afcc-aa1410815d17-tls-certs\") pod \"kube-auth-proxy-85ccdd577f-llvhl\" (UID: \"01362f25-65e7-4ec6-afcc-aa1410815d17\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" Nov 02 12:02:41.686290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.686224 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01362f25-65e7-4ec6-afcc-aa1410815d17-tmp\") pod \"kube-auth-proxy-85ccdd577f-llvhl\" (UID: \"01362f25-65e7-4ec6-afcc-aa1410815d17\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" Nov 02 12:02:41.686562 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.686545 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qt4d\" (UniqueName: \"kubernetes.io/projected/01362f25-65e7-4ec6-afcc-aa1410815d17-kube-api-access-6qt4d\") pod \"kube-auth-proxy-85ccdd577f-llvhl\" (UID: \"01362f25-65e7-4ec6-afcc-aa1410815d17\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" Nov 02 12:02:41.692088 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.692058 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01362f25-65e7-4ec6-afcc-aa1410815d17-tls-certs\") pod \"kube-auth-proxy-85ccdd577f-llvhl\" (UID: \"01362f25-65e7-4ec6-afcc-aa1410815d17\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" Nov 02 12:02:41.693629 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.693606 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01362f25-65e7-4ec6-afcc-aa1410815d17-tmp\") pod \"kube-auth-proxy-85ccdd577f-llvhl\" (UID: \"01362f25-65e7-4ec6-afcc-aa1410815d17\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" Nov 02 12:02:41.769433 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.769410 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qt4d\" (UniqueName: \"kubernetes.io/projected/01362f25-65e7-4ec6-afcc-aa1410815d17-kube-api-access-6qt4d\") pod \"kube-auth-proxy-85ccdd577f-llvhl\" (UID: \"01362f25-65e7-4ec6-afcc-aa1410815d17\") " pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" Nov 02 12:02:41.783483 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.783458 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:41.865957 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:41.865934 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" Nov 02 12:02:42.015780 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:42.015734 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-ck9gr" podStartSLOduration=1.484400163 podStartE2EDuration="7.015717746s" totalImagesPullingTime="5.531317583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:02:42.015055065 +0000 UTC m=+379.045291317" watchObservedRunningTime="2026-11-02 12:02:42.015717746 +0000 UTC m=+379.045953996" Nov 02 12:02:42.046807 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:42.046779 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:42.235959 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:42.235828 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl"] Nov 02 12:02:42.238633 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:02:42.238608 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01362f25_65e7_4ec6_afcc_aa1410815d17.slice/crio-890f9c6af848a3e5393c51def395e7692d73c0fa6cc186ef5cc4bb36def78fd6 WatchSource:0}: Error finding container 890f9c6af848a3e5393c51def395e7692d73c0fa6cc186ef5cc4bb36def78fd6: Status 404 returned error can't find the container with id 890f9c6af848a3e5393c51def395e7692d73c0fa6cc186ef5cc4bb36def78fd6 Nov 02 12:02:42.377140 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:42.377113 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:42.590156 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:42.590098 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" event={"ID":"01362f25-65e7-4ec6-afcc-aa1410815d17","Type":"ContainerStarted","Data":"890f9c6af848a3e5393c51def395e7692d73c0fa6cc186ef5cc4bb36def78fd6"} Nov 02 12:02:42.645577 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:42.645553 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:42.928211 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:42.928188 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:43.295043 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:43.294960 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:45.559960 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:45.559929 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-wqxtt" Nov 02 12:02:45.604799 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:45.604767 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" event={"ID":"01362f25-65e7-4ec6-afcc-aa1410815d17","Type":"ContainerStarted","Data":"5c22bdafdfa091eda4b19a86226e55f030385352a5fdabd492ffb72715110c22"} Nov 02 12:02:45.797823 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:45.797777 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-85ccdd577f-llvhl" podStartSLOduration=1.984871496 podStartE2EDuration="4.797764227s" totalImagesPullingTime="2.812892731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:02:45.795917795 +0000 UTC m=+382.826154046" watchObservedRunningTime="2026-11-02 12:02:45.797764227 +0000 UTC m=+382.828000477" Nov 02 12:02:47.431195 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.431148 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-vq24k"] Nov 02 12:02:47.434535 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.434512 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vq24k" Nov 02 12:02:47.534655 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.534618 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2pd9\" (UniqueName: \"kubernetes.io/projected/2c5082af-ba97-4504-91f3-95a47f90b177-kube-api-access-b2pd9\") pod \"cert-manager-597b95cb84-vq24k\" (UID: \"2c5082af-ba97-4504-91f3-95a47f90b177\") " pod="cert-manager/cert-manager-597b95cb84-vq24k" Nov 02 12:02:47.534972 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.534953 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c5082af-ba97-4504-91f3-95a47f90b177-bound-sa-token\") pod \"cert-manager-597b95cb84-vq24k\" (UID: \"2c5082af-ba97-4504-91f3-95a47f90b177\") " pod="cert-manager/cert-manager-597b95cb84-vq24k" Nov 02 12:02:47.547674 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.547650 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ns8w6\"" Nov 02 12:02:47.549019 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.548996 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vq24k"] Nov 02 12:02:47.636305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.636272 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2pd9\" (UniqueName: \"kubernetes.io/projected/2c5082af-ba97-4504-91f3-95a47f90b177-kube-api-access-b2pd9\") pod \"cert-manager-597b95cb84-vq24k\" (UID: \"2c5082af-ba97-4504-91f3-95a47f90b177\") " pod="cert-manager/cert-manager-597b95cb84-vq24k" Nov 02 12:02:47.636305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.636308 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c5082af-ba97-4504-91f3-95a47f90b177-bound-sa-token\") pod \"cert-manager-597b95cb84-vq24k\" (UID: \"2c5082af-ba97-4504-91f3-95a47f90b177\") " pod="cert-manager/cert-manager-597b95cb84-vq24k" Nov 02 12:02:47.697177 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.697111 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c5082af-ba97-4504-91f3-95a47f90b177-bound-sa-token\") pod \"cert-manager-597b95cb84-vq24k\" (UID: \"2c5082af-ba97-4504-91f3-95a47f90b177\") " pod="cert-manager/cert-manager-597b95cb84-vq24k" Nov 02 12:02:47.697177 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.697119 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2pd9\" (UniqueName: \"kubernetes.io/projected/2c5082af-ba97-4504-91f3-95a47f90b177-kube-api-access-b2pd9\") pod \"cert-manager-597b95cb84-vq24k\" (UID: \"2c5082af-ba97-4504-91f3-95a47f90b177\") " pod="cert-manager/cert-manager-597b95cb84-vq24k" Nov 02 12:02:47.745925 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.745902 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vq24k" Nov 02 12:02:47.928269 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:47.928239 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:48.118525 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:48.118433 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vq24k"] Nov 02 12:02:48.121728 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:02:48.121702 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c5082af_ba97_4504_91f3_95a47f90b177.slice/crio-a9fd63b1a608843c665fe06c96b6834064fdbcad385966d43cb3872973cb13fe WatchSource:0}: Error finding container a9fd63b1a608843c665fe06c96b6834064fdbcad385966d43cb3872973cb13fe: Status 404 returned error can't find the container with id a9fd63b1a608843c665fe06c96b6834064fdbcad385966d43cb3872973cb13fe Nov 02 12:02:48.276515 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:48.276466 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:48.616800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:48.616766 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vq24k" event={"ID":"2c5082af-ba97-4504-91f3-95a47f90b177","Type":"ContainerStarted","Data":"cd21fe8084949dc40cc1698cbb5c6b78c9d9ebc1e4ec1459860bfb299681586d"} Nov 02 12:02:48.617166 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:48.616808 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vq24k" event={"ID":"2c5082af-ba97-4504-91f3-95a47f90b177","Type":"ContainerStarted","Data":"a9fd63b1a608843c665fe06c96b6834064fdbcad385966d43cb3872973cb13fe"} Nov 02 12:02:48.733368 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:48.733317 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-vq24k" podStartSLOduration=1.7333040130000001 podStartE2EDuration="1.733304013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:02:48.732480145 +0000 UTC m=+385.762716395" watchObservedRunningTime="2026-11-02 12:02:48.733304013 +0000 UTC m=+385.763540262" Nov 02 12:02:49.274923 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:49.274893 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:50.455106 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:50.455068 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:50.715903 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:50.715826 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:52.921905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:52.921876 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:53.173054 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:53.172977 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:58.071266 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:58.071223 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:58.377696 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:02:58.377607 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:02:59.372585 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:02:59.372550 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:03:05.608546 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:05.608463 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:06.942004 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:06.941968 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:07.211546 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:07.211480 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:07.720122 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:07.720089 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:09.525307 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:09.525277 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:09.833239 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:09.833127 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:12.921924 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:12.921881 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:13.168984 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:13.168952 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:15.729313 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:15.729279 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:15.974731 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:15.974696 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:18.307546 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:18.307509 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:18.554450 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:18.554418 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:18.974656 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:18.974618 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:19.226326 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:19.226249 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:19.511092 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:19.511001 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:19.764653 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:19.764581 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:27.213592 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:27.213556 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:27.566948 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:27.566853 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:28.889337 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:28.889302 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:29.149478 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:29.149401 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:29.372112 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:03:29.372080 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:03:29.401469 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:29.401396 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:30.518696 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:30.518660 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:30.768306 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:30.768271 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:37.057085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:37.057052 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:37.314924 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:37.314849 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:40.262186 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:40.262150 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:40.550678 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:03:40.550601 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:03:59.372266 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:03:59.372228 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:04:05.953948 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:05.953909 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:10.290369 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:10.290331 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:10.563965 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:10.563896 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:12.381966 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:12.381928 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:12.644617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:12.644537 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:12.887232 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:12.887201 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:13.410888 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.410858 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g"] Nov 02 12:04:13.412986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.412970 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.448273 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.448246 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-openshift-service-ca\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.448382 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.448276 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-maas-api-tls\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.448382 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.448369 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdwhq\" (UniqueName: \"kubernetes.io/projected/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-kube-api-access-mdwhq\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.448453 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.448414 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-metrics-certs\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.514560 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.514530 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 12:04:13.514685 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.514568 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 12:04:13.514685 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.514609 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-pl94d\"" Nov 02 12:04:13.514685 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.514661 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 12:04:13.514834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.514727 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 12:04:13.515995 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.515976 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g"] Nov 02 12:04:13.549192 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.549172 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-openshift-service-ca\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.549315 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.549196 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-maas-api-tls\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.549315 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.549229 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdwhq\" (UniqueName: \"kubernetes.io/projected/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-kube-api-access-mdwhq\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.549315 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.549276 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-metrics-certs\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.549884 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.549865 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-openshift-service-ca\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.551697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.551676 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-metrics-certs\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.551786 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.551726 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-maas-api-tls\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.609925 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.609907 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdwhq\" (UniqueName: \"kubernetes.io/projected/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-kube-api-access-mdwhq\") pod \"maas-api-cd478b64b-p5v5g\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:13.723923 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:13.723903 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:14.090150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:14.090126 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g"] Nov 02 12:04:14.092516 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:04:14.092473 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9d52ee_1ef1_4bb0_a6cf_21d5090bb0ea.slice/crio-8bc4d18600a583e7776e4785649d72d5d66a8683bc7fdf53e6817f0384de81f8 WatchSource:0}: Error finding container 8bc4d18600a583e7776e4785649d72d5d66a8683bc7fdf53e6817f0384de81f8: Status 404 returned error can't find the container with id 8bc4d18600a583e7776e4785649d72d5d66a8683bc7fdf53e6817f0384de81f8 Nov 02 12:04:14.978743 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:14.978712 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" event={"ID":"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea","Type":"ContainerStarted","Data":"8bc4d18600a583e7776e4785649d72d5d66a8683bc7fdf53e6817f0384de81f8"} Nov 02 12:04:15.542201 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:15.542169 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:15.805056 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:15.804952 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:16.054438 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.054398 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:16.308352 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.308322 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:16.441211 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.441181 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp"] Nov 02 12:04:16.444113 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.444090 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:04:16.467890 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.467863 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbxs7\" (UniqueName: \"kubernetes.io/projected/ffca5ead-f6f4-46f5-8b40-ee050f859e83-kube-api-access-rbxs7\") pod \"llmisvc-controller-manager-869d5dcc5d-lftbp\" (UID: \"ffca5ead-f6f4-46f5-8b40-ee050f859e83\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:04:16.468008 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.467914 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffca5ead-f6f4-46f5-8b40-ee050f859e83-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lftbp\" (UID: \"ffca5ead-f6f4-46f5-8b40-ee050f859e83\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:04:16.496475 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.496451 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 12:04:16.545590 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.545568 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 12:04:16.546211 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.546190 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2qvgc\"" Nov 02 12:04:16.546211 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.546200 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 12:04:16.547209 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.547192 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp"] Nov 02 12:04:16.560323 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.560276 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7w4zh"] Nov 02 12:04:16.564543 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.564490 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" Nov 02 12:04:16.565957 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.565924 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:16.570739 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.570713 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ced755a-4baf-4b4b-96df-f8a57839c593-tls-certs\") pod \"model-serving-api-75bd89fdfc-7w4zh\" (UID: \"5ced755a-4baf-4b4b-96df-f8a57839c593\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" Nov 02 12:04:16.571234 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.571077 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbxs7\" (UniqueName: \"kubernetes.io/projected/ffca5ead-f6f4-46f5-8b40-ee050f859e83-kube-api-access-rbxs7\") pod \"llmisvc-controller-manager-869d5dcc5d-lftbp\" (UID: \"ffca5ead-f6f4-46f5-8b40-ee050f859e83\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:04:16.571234 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.571129 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhxm\" (UniqueName: \"kubernetes.io/projected/5ced755a-4baf-4b4b-96df-f8a57839c593-kube-api-access-jxhxm\") pod \"model-serving-api-75bd89fdfc-7w4zh\" (UID: \"5ced755a-4baf-4b4b-96df-f8a57839c593\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" Nov 02 12:04:16.571234 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.571160 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffca5ead-f6f4-46f5-8b40-ee050f859e83-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lftbp\" (UID: \"ffca5ead-f6f4-46f5-8b40-ee050f859e83\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:04:16.575085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.575064 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffca5ead-f6f4-46f5-8b40-ee050f859e83-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-lftbp\" (UID: \"ffca5ead-f6f4-46f5-8b40-ee050f859e83\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:04:16.623961 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.623935 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7w4zh"] Nov 02 12:04:16.637327 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.637307 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbxs7\" (UniqueName: \"kubernetes.io/projected/ffca5ead-f6f4-46f5-8b40-ee050f859e83-kube-api-access-rbxs7\") pod \"llmisvc-controller-manager-869d5dcc5d-lftbp\" (UID: \"ffca5ead-f6f4-46f5-8b40-ee050f859e83\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:04:16.637865 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.637845 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pmj6p\"" Nov 02 12:04:16.637947 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.637925 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 12:04:16.671700 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.671671 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhxm\" (UniqueName: \"kubernetes.io/projected/5ced755a-4baf-4b4b-96df-f8a57839c593-kube-api-access-jxhxm\") pod \"model-serving-api-75bd89fdfc-7w4zh\" (UID: \"5ced755a-4baf-4b4b-96df-f8a57839c593\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" Nov 02 12:04:16.671806 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.671728 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ced755a-4baf-4b4b-96df-f8a57839c593-tls-certs\") pod \"model-serving-api-75bd89fdfc-7w4zh\" (UID: \"5ced755a-4baf-4b4b-96df-f8a57839c593\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" Nov 02 12:04:16.674330 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.674305 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ced755a-4baf-4b4b-96df-f8a57839c593-tls-certs\") pod \"model-serving-api-75bd89fdfc-7w4zh\" (UID: \"5ced755a-4baf-4b4b-96df-f8a57839c593\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" Nov 02 12:04:16.751689 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.751659 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhxm\" (UniqueName: \"kubernetes.io/projected/5ced755a-4baf-4b4b-96df-f8a57839c593-kube-api-access-jxhxm\") pod \"model-serving-api-75bd89fdfc-7w4zh\" (UID: \"5ced755a-4baf-4b4b-96df-f8a57839c593\") " pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" Nov 02 12:04:16.756295 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.756270 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:04:16.830158 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.830135 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:16.887652 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:16.887626 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" Nov 02 12:04:17.139155 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:17.139069 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp"] Nov 02 12:04:17.142390 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:04:17.142361 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podffca5ead_f6f4_46f5_8b40_ee050f859e83.slice/crio-fa2eb1d8bd60455b3e01ac7cb9003b3682d92aecc722da8d63d52cba5d957596 WatchSource:0}: Error finding container fa2eb1d8bd60455b3e01ac7cb9003b3682d92aecc722da8d63d52cba5d957596: Status 404 returned error can't find the container with id fa2eb1d8bd60455b3e01ac7cb9003b3682d92aecc722da8d63d52cba5d957596 Nov 02 12:04:17.257778 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:17.257669 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7w4zh"] Nov 02 12:04:17.260172 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:04:17.260152 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ced755a_4baf_4b4b_96df_f8a57839c593.slice/crio-0b27e32ce85793613d6deb3e6b367af83d4a08b73735c023ecf523cf4ccb8c5a WatchSource:0}: Error finding container 0b27e32ce85793613d6deb3e6b367af83d4a08b73735c023ecf523cf4ccb8c5a: Status 404 returned error can't find the container with id 0b27e32ce85793613d6deb3e6b367af83d4a08b73735c023ecf523cf4ccb8c5a Nov 02 12:04:17.802512 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:17.802471 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:17.993161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:17.993131 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" event={"ID":"ffca5ead-f6f4-46f5-8b40-ee050f859e83","Type":"ContainerStarted","Data":"fa2eb1d8bd60455b3e01ac7cb9003b3682d92aecc722da8d63d52cba5d957596"} Nov 02 12:04:17.994233 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:17.994211 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" event={"ID":"5ced755a-4baf-4b4b-96df-f8a57839c593","Type":"ContainerStarted","Data":"0b27e32ce85793613d6deb3e6b367af83d4a08b73735c023ecf523cf4ccb8c5a"} Nov 02 12:04:20.004795 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:20.004761 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" event={"ID":"ffca5ead-f6f4-46f5-8b40-ee050f859e83","Type":"ContainerStarted","Data":"2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab"} Nov 02 12:04:20.005262 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:20.004832 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:04:20.006365 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:20.006344 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" event={"ID":"5ced755a-4baf-4b4b-96df-f8a57839c593","Type":"ContainerStarted","Data":"1d165b8dc22ba2931cd06a1b9c500571bef50d656b1656b01eb507b14bc9569a"} Nov 02 12:04:20.006468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:20.006391 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" Nov 02 12:04:20.121521 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:20.121177 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" podStartSLOduration=1.73219807 podStartE2EDuration="4.121163038s" totalImagesPullingTime="2.388964968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:04:20.119528454 +0000 UTC m=+477.149764701" watchObservedRunningTime="2026-11-02 12:04:20.121163038 +0000 UTC m=+477.151399288" Nov 02 12:04:20.237179 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:20.237129 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" podStartSLOduration=2.052182306 podStartE2EDuration="4.237112077s" totalImagesPullingTime="2.184929771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:04:20.235618945 +0000 UTC m=+477.265855195" watchObservedRunningTime="2026-11-02 12:04:20.237112077 +0000 UTC m=+477.267348328" Nov 02 12:04:21.012843 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:21.012802 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" event={"ID":"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea","Type":"ContainerStarted","Data":"7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0"} Nov 02 12:04:21.013251 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:21.013157 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:21.189076 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:21.189034 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" podStartSLOduration=1.4192229699999999 podStartE2EDuration="8.189020358s" totalImagesPullingTime="6.769797388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:04:21.187591759 +0000 UTC m=+478.217828010" watchObservedRunningTime="2026-11-02 12:04:21.189020358 +0000 UTC m=+478.219256658" Nov 02 12:04:22.988175 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:22.988143 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:23.236724 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:23.236694 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:23.487024 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:23.486976 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:23.728693 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:23.728665 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:23.977790 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:23.977756 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:24.228131 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:24.228049 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:26.722316 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:26.722274 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:26.975632 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:26.975544 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:28.026812 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:28.026777 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:28.657896 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:28.657859 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:28.933417 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:28.933374 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:04:29.065359 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.065325 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd"] Nov 02 12:04:29.068888 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.068860 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.156367 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.156333 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-maas-api-tls\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.156484 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.156451 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lr9j\" (UniqueName: \"kubernetes.io/projected/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-kube-api-access-8lr9j\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.156484 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.156481 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-metrics-certs\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.156611 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.156590 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-openshift-service-ca\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.172037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.172015 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd"] Nov 02 12:04:29.257386 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.257332 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-maas-api-tls\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.257478 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.257383 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lr9j\" (UniqueName: \"kubernetes.io/projected/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-kube-api-access-8lr9j\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.257478 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.257413 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-metrics-certs\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.257478 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.257445 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-openshift-service-ca\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.258131 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.258114 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-openshift-service-ca\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.259856 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.259836 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-maas-api-tls\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.259980 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.259961 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-metrics-certs\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.317940 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.317916 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lr9j\" (UniqueName: \"kubernetes.io/projected/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-kube-api-access-8lr9j\") pod \"maas-api-6866d668fb-cfdrd\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.371236 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:04:29.371201 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:04:29.382455 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.382437 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:29.754739 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:29.754716 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd"] Nov 02 12:04:29.756238 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:04:29.756213 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32dda5be_5c7f_41f4_bef6_8eff5d1003fb.slice/crio-23dbb72b95c2837dab87a16352ddaa96f3a0a39e5ee8e8f7d52636291b06949a WatchSource:0}: Error finding container 23dbb72b95c2837dab87a16352ddaa96f3a0a39e5ee8e8f7d52636291b06949a: Status 404 returned error can't find the container with id 23dbb72b95c2837dab87a16352ddaa96f3a0a39e5ee8e8f7d52636291b06949a Nov 02 12:04:30.058216 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:30.058143 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" event={"ID":"32dda5be-5c7f-41f4-bef6-8eff5d1003fb","Type":"ContainerStarted","Data":"23dbb72b95c2837dab87a16352ddaa96f3a0a39e5ee8e8f7d52636291b06949a"} Nov 02 12:04:31.017228 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:31.017206 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-7w4zh" Nov 02 12:04:31.065377 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:31.065346 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" event={"ID":"32dda5be-5c7f-41f4-bef6-8eff5d1003fb","Type":"ContainerStarted","Data":"b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f"} Nov 02 12:04:31.065519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:31.065412 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:31.249581 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:31.249527 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" podStartSLOduration=1.453444361 podStartE2EDuration="2.249513309s" totalImagesPullingTime="796.068948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:04:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:04:31.249243508 +0000 UTC m=+488.279479758" watchObservedRunningTime="2026-11-02 12:04:31.249513309 +0000 UTC m=+488.279749553" Nov 02 12:04:37.075004 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.074931 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 12:04:37.245256 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.243650 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g"] Nov 02 12:04:37.246111 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.246057 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" podUID="ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea" containerName="maas-api" containerID="cri-o://7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0" gracePeriod=30 Nov 02 12:04:37.492157 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.492130 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:37.520391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.520369 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-metrics-certs\") pod \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " Nov 02 12:04:37.520466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.520400 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-maas-api-tls\") pod \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " Nov 02 12:04:37.520466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.520420 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-kube-api-access-mdwhq\" (UniqueName: \"kubernetes.io/projected/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-kube-api-access-mdwhq\") pod \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " Nov 02 12:04:37.520466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.520444 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-openshift-service-ca\") pod \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\" (UID: \"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea\") " Nov 02 12:04:37.520833 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.520809 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-openshift-service-ca" pod "ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea" (UID: "ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:04:37.522434 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.522412 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-maas-api-tls" pod "ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea" (UID: "ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:04:37.522533 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.522488 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-kube-api-access-mdwhq" pod "ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea" (UID: "ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea"). InnerVolumeSpecName "kube-api-access-mdwhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:04:37.522580 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.522546 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-metrics-certs" pod "ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea" (UID: "ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:04:37.621466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.621401 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-metrics-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:04:37.621466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.621437 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-maas-api-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:04:37.621466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.621454 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdwhq\" (UniqueName: \"kubernetes.io/projected/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-kube-api-access-mdwhq\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:04:37.621634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:37.621477 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea-openshift-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:04:38.095554 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:38.095489 4312 generic.go:356] "Generic (PLEG): container finished" podID="ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea" containerID="7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0" exitCode=0 Nov 02 12:04:38.095918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:38.095570 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" Nov 02 12:04:38.095918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:38.095585 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" event={"ID":"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea","Type":"ContainerDied","Data":"7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0"} Nov 02 12:04:38.095918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:38.095642 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g" event={"ID":"ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea","Type":"ContainerDied","Data":"8bc4d18600a583e7776e4785649d72d5d66a8683bc7fdf53e6817f0384de81f8"} Nov 02 12:04:38.095918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:38.095667 4312 scope.go:122] "RemoveContainer" containerID="7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0" Nov 02 12:04:38.106684 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:38.106665 4312 scope.go:122] "RemoveContainer" containerID="7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0" Nov 02 12:04:38.107029 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:04:38.107008 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0\": container with ID starting with 7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0 not found: ID does not exist" containerID="7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0" Nov 02 12:04:38.107093 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:38.107040 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0"} err="failed to get container status \"7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0\": rpc error: code = NotFound desc = could not find container \"7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0\": container with ID starting with 7031b4a439856d10d9e5a979e15c56f10690569f052cf6f1d541fa92c3a303d0 not found: ID does not exist" Nov 02 12:04:38.259015 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:38.258987 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g"] Nov 02 12:04:38.308366 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:38.308346 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-cd478b64b-p5v5g"] Nov 02 12:04:39.074434 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:39.074398 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea" path="/var/lib/kubelet/pods/ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea/volumes" Nov 02 12:04:51.015888 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:04:51.015856 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:04:59.372079 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:04:59.372041 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:05:06.292089 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:06.292046 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:05:06.541800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:06.541727 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:05:16.243824 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:16.243788 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:05:29.373247 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:05:29.373205 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:05:33.982334 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:33.982298 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz"] Nov 02 12:05:33.982874 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:33.982854 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea" containerName="maas-api" Nov 02 12:05:33.982951 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:33.982876 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea" containerName="maas-api" Nov 02 12:05:33.983003 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:33.982964 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9d52ee-1ef1-4bb0-a6cf-21d5090bb0ea" containerName="maas-api" Nov 02 12:05:33.985547 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:33.985528 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" Nov 02 12:05:34.045506 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:34.045470 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz"] Nov 02 12:05:34.139565 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:34.139542 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdf81983-c2ed-4444-9837-16ffca122340-cert\") pod \"llmisvc-controller-manager-6466c554d8-vfxbz\" (UID: \"fdf81983-c2ed-4444-9837-16ffca122340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" Nov 02 12:05:34.139658 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:34.139605 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxz27\" (UniqueName: \"kubernetes.io/projected/fdf81983-c2ed-4444-9837-16ffca122340-kube-api-access-fxz27\") pod \"llmisvc-controller-manager-6466c554d8-vfxbz\" (UID: \"fdf81983-c2ed-4444-9837-16ffca122340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" Nov 02 12:05:34.240967 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:34.240901 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxz27\" (UniqueName: \"kubernetes.io/projected/fdf81983-c2ed-4444-9837-16ffca122340-kube-api-access-fxz27\") pod \"llmisvc-controller-manager-6466c554d8-vfxbz\" (UID: \"fdf81983-c2ed-4444-9837-16ffca122340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" Nov 02 12:05:34.240967 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:34.240958 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdf81983-c2ed-4444-9837-16ffca122340-cert\") pod \"llmisvc-controller-manager-6466c554d8-vfxbz\" (UID: \"fdf81983-c2ed-4444-9837-16ffca122340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" Nov 02 12:05:34.243683 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:34.243657 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fdf81983-c2ed-4444-9837-16ffca122340-cert\") pod \"llmisvc-controller-manager-6466c554d8-vfxbz\" (UID: \"fdf81983-c2ed-4444-9837-16ffca122340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" Nov 02 12:05:34.302233 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:34.302207 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxz27\" (UniqueName: \"kubernetes.io/projected/fdf81983-c2ed-4444-9837-16ffca122340-kube-api-access-fxz27\") pod \"llmisvc-controller-manager-6466c554d8-vfxbz\" (UID: \"fdf81983-c2ed-4444-9837-16ffca122340\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" Nov 02 12:05:34.597294 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:34.597202 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" Nov 02 12:05:34.967086 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:34.967057 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz"] Nov 02 12:05:34.969381 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:05:34.969343 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf81983_c2ed_4444_9837_16ffca122340.slice/crio-9a260029b5ce792c4e5a26c168de5fef5ddc821ecb02d2fd83ff563ad0441fb4 WatchSource:0}: Error finding container 9a260029b5ce792c4e5a26c168de5fef5ddc821ecb02d2fd83ff563ad0441fb4: Status 404 returned error can't find the container with id 9a260029b5ce792c4e5a26c168de5fef5ddc821ecb02d2fd83ff563ad0441fb4 Nov 02 12:05:35.337343 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:35.337270 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" event={"ID":"fdf81983-c2ed-4444-9837-16ffca122340","Type":"ContainerStarted","Data":"9a260029b5ce792c4e5a26c168de5fef5ddc821ecb02d2fd83ff563ad0441fb4"} Nov 02 12:05:36.344155 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:36.344125 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" event={"ID":"fdf81983-c2ed-4444-9837-16ffca122340","Type":"ContainerStarted","Data":"6e9132b7ce859d9057c01acdebf1b4fb4ece34be46702632fb2efc6f6a2ec58b"} Nov 02 12:05:36.344530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:36.344186 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" Nov 02 12:05:36.461148 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:36.461104 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" podStartSLOduration=2.678700345 podStartE2EDuration="3.461089599s" totalImagesPullingTime="782.389254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:05:36.459645265 +0000 UTC m=+553.489881515" watchObservedRunningTime="2026-11-02 12:05:36.461089599 +0000 UTC m=+553.491325849" Nov 02 12:05:36.709822 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:36.709793 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:05:57.046152 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:05:57.046113 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:05:59.373679 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:05:59.373576 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:06:06.652261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:06.652115 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:07.350769 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:07.350732 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vfxbz" Nov 02 12:06:07.516935 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:07.516905 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp"] Nov 02 12:06:07.517204 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:07.517174 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" podUID="ffca5ead-f6f4-46f5-8b40-ee050f859e83" containerName="manager" containerID="cri-o://2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab" gracePeriod=30 Nov 02 12:06:07.767511 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:07.767476 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:06:07.889441 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:07.889373 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffca5ead-f6f4-46f5-8b40-ee050f859e83-cert\" (UniqueName: \"kubernetes.io/secret/ffca5ead-f6f4-46f5-8b40-ee050f859e83-cert\") pod \"ffca5ead-f6f4-46f5-8b40-ee050f859e83\" (UID: \"ffca5ead-f6f4-46f5-8b40-ee050f859e83\") " Nov 02 12:06:07.889441 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:07.889410 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffca5ead-f6f4-46f5-8b40-ee050f859e83-kube-api-access-rbxs7\" (UniqueName: \"kubernetes.io/projected/ffca5ead-f6f4-46f5-8b40-ee050f859e83-kube-api-access-rbxs7\") pod \"ffca5ead-f6f4-46f5-8b40-ee050f859e83\" (UID: \"ffca5ead-f6f4-46f5-8b40-ee050f859e83\") " Nov 02 12:06:07.891616 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:07.891581 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffca5ead-f6f4-46f5-8b40-ee050f859e83-cert" pod "ffca5ead-f6f4-46f5-8b40-ee050f859e83" (UID: "ffca5ead-f6f4-46f5-8b40-ee050f859e83"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:06:07.891711 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:07.891640 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffca5ead-f6f4-46f5-8b40-ee050f859e83-kube-api-access-rbxs7" pod "ffca5ead-f6f4-46f5-8b40-ee050f859e83" (UID: "ffca5ead-f6f4-46f5-8b40-ee050f859e83"). InnerVolumeSpecName "kube-api-access-rbxs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:06:07.990968 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:07.990945 4312 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffca5ead-f6f4-46f5-8b40-ee050f859e83-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:06:07.990968 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:07.990967 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbxs7\" (UniqueName: \"kubernetes.io/projected/ffca5ead-f6f4-46f5-8b40-ee050f859e83-kube-api-access-rbxs7\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:06:08.483774 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:08.483735 4312 generic.go:356] "Generic (PLEG): container finished" podID="ffca5ead-f6f4-46f5-8b40-ee050f859e83" containerID="2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab" exitCode=0 Nov 02 12:06:08.483968 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:08.483801 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" Nov 02 12:06:08.483968 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:08.483815 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" event={"ID":"ffca5ead-f6f4-46f5-8b40-ee050f859e83","Type":"ContainerDied","Data":"2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab"} Nov 02 12:06:08.483968 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:08.483859 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp" event={"ID":"ffca5ead-f6f4-46f5-8b40-ee050f859e83","Type":"ContainerDied","Data":"fa2eb1d8bd60455b3e01ac7cb9003b3682d92aecc722da8d63d52cba5d957596"} Nov 02 12:06:08.483968 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:08.483884 4312 scope.go:122] "RemoveContainer" containerID="2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab" Nov 02 12:06:08.493457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:08.493428 4312 scope.go:122] "RemoveContainer" containerID="2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab" Nov 02 12:06:08.493840 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:06:08.493818 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab\": container with ID starting with 2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab not found: ID does not exist" containerID="2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab" Nov 02 12:06:08.493928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:08.493849 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab"} err="failed to get container status \"2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab\": rpc error: code = NotFound desc = could not find container \"2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab\": container with ID starting with 2b4a105462ea8c82a2f066e3f0d9e5e4dd47577c3a65986da768281022bbd4ab not found: ID does not exist" Nov 02 12:06:08.648566 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:08.648535 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp"] Nov 02 12:06:08.697778 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:08.697744 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-lftbp"] Nov 02 12:06:09.074053 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:09.074021 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffca5ead-f6f4-46f5-8b40-ee050f859e83" path="/var/lib/kubelet/pods/ffca5ead-f6f4-46f5-8b40-ee050f859e83/volumes" Nov 02 12:06:09.440856 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:09.440821 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:09.757709 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:09.757625 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:12.452766 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:12.452729 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:17.545710 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:17.545680 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:17.799892 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:17.799810 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:29.372788 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:06:29.372744 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:06:40.171859 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:40.171817 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:40.544809 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:40.544700 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:41.772835 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:41.772790 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:42.031471 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:42.031389 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:46.660606 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:46.660575 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:46.921181 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:46.921075 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:49.395157 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:49.395082 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:55.509414 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:06:55.509377 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:06:59.372782 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:06:59.372750 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:07:00.410866 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:00.410831 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:04.480684 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:04.480649 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:06.596730 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:06.596696 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:06.994456 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:06.994431 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:07.888038 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:07.888002 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:12.605715 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:12.605681 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:13.452600 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.452564 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb"] Nov 02 12:07:13.452916 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.452904 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ffca5ead-f6f4-46f5-8b40-ee050f859e83" containerName="manager" Nov 02 12:07:13.452966 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.452917 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffca5ead-f6f4-46f5-8b40-ee050f859e83" containerName="manager" Nov 02 12:07:13.453000 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.452989 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffca5ead-f6f4-46f5-8b40-ee050f859e83" containerName="manager" Nov 02 12:07:13.455761 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.455746 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.480955 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.480930 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.481077 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.481006 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.481077 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.481059 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.481149 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.481077 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.481149 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.481094 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.481212 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.481146 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/684df0a9-463d-4c13-bbcd-33150e90687d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.481212 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.481167 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/684df0a9-463d-4c13-bbcd-33150e90687d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.481276 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.481207 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfvj7\" (UniqueName: \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-kube-api-access-dfvj7\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.481276 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.481271 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.507569 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.507546 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c5ldd\"" Nov 02 12:07:13.557065 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.557036 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 12:07:13.559550 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.559530 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb"] Nov 02 12:07:13.582531 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.582489 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.582637 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.582550 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.582697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.582653 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.582697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.582680 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.582810 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.582706 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/684df0a9-463d-4c13-bbcd-33150e90687d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.582810 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.582730 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/684df0a9-463d-4c13-bbcd-33150e90687d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.582810 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.582782 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfvj7\" (UniqueName: \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-kube-api-access-dfvj7\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.582963 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.582817 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.582963 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.582829 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.582963 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.582858 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.582963 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.582906 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.583132 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.583004 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.583132 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.583073 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.583422 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.583397 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/684df0a9-463d-4c13-bbcd-33150e90687d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.585023 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.584999 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.585250 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.585228 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/684df0a9-463d-4c13-bbcd-33150e90687d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.648582 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.648559 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfvj7\" (UniqueName: \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-kube-api-access-dfvj7\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.666619 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.666597 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cjqgb\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:13.768105 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:13.768044 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:14.191833 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:14.191810 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb"] Nov 02 12:07:14.194137 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:07:14.194113 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684df0a9_463d_4c13_bbcd_33150e90687d.slice/crio-8c92de8ed6b4a689ac85d96654e62814580f7a5a4eb7afba4a0511bb0d33628a WatchSource:0}: Error finding container 8c92de8ed6b4a689ac85d96654e62814580f7a5a4eb7afba4a0511bb0d33628a: Status 404 returned error can't find the container with id 8c92de8ed6b4a689ac85d96654e62814580f7a5a4eb7afba4a0511bb0d33628a Nov 02 12:07:14.196068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:14.196051 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 12:07:14.763925 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:14.763886 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" event={"ID":"684df0a9-463d-4c13-bbcd-33150e90687d","Type":"ContainerStarted","Data":"8c92de8ed6b4a689ac85d96654e62814580f7a5a4eb7afba4a0511bb0d33628a"} Nov 02 12:07:16.029749 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:16.029711 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:18.059127 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:18.059097 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:07:18.059413 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:18.059160 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:07:18.059413 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:18.059187 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:07:18.785302 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:18.785268 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" event={"ID":"684df0a9-463d-4c13-bbcd-33150e90687d","Type":"ContainerStarted","Data":"13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d"} Nov 02 12:07:19.768793 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:19.768762 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:19.772617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:19.772578 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cjqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 12:07:19.772757 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:19.772643 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 12:07:20.702924 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:20.702890 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:20.770517 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:20.769835 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cjqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 12:07:20.770517 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:20.769901 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 12:07:21.768873 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:21.768837 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cjqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 12:07:21.769140 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:21.769113 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 12:07:22.768877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:22.768839 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cjqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 12:07:22.769463 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:22.769424 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 12:07:23.772520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:23.768841 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cjqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 12:07:23.772520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:23.768902 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 12:07:24.769141 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:24.769105 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cjqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 02 12:07:24.769303 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:24.769151 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 02 12:07:26.769284 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:26.769254 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cjqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 12:07:26.769712 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:26.769302 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 12:07:27.810682 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:27.810641 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cjqgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 12:07:27.811090 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:27.810700 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 12:07:28.536543 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.536514 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:28.536826 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.536808 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:28.537525 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.537485 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:07:28.581954 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.581899 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" podStartSLOduration=11.719856951 podStartE2EDuration="15.581881127s" totalImagesPullingTime="3.862024176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:07:18.95451898 +0000 UTC m=+655.984755228" watchObservedRunningTime="2026-11-02 12:07:28.581881127 +0000 UTC m=+665.612117378" Nov 02 12:07:28.583775 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.583753 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng"] Nov 02 12:07:28.587868 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.587849 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.682702 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.682669 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng"] Nov 02 12:07:28.734586 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.734558 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.734738 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.734619 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.734738 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.734662 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f4312e-6933-473d-ace6-29cb4b45512d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.734827 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.734738 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.734827 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.734756 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.734827 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.734771 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.734928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.734841 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.734928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.734890 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.735017 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.735001 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8dgw\" (UniqueName: \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-kube-api-access-b8dgw\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.835458 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.835382 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8dgw\" (UniqueName: \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-kube-api-access-b8dgw\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.835458 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.835436 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.835479 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.835521 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f4312e-6933-473d-ace6-29cb4b45512d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.835573 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.835600 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.835627 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.835659 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.835705 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.835945 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836413 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.836293 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f4312e-6933-473d-ace6-29cb4b45512d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836413 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.836345 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836577 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.836554 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.836639 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.836622 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.838416 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.838384 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.838765 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.838747 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.900571 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.900544 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8dgw\" (UniqueName: \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-kube-api-access-b8dgw\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:28.927866 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:28.927839 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-th4ng\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:29.200896 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:29.200869 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:29.371533 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:07:29.371478 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:07:29.624957 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:29.624935 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng"] Nov 02 12:07:29.628589 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:07:29.628559 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f4312e_6933_473d_ace6_29cb4b45512d.slice/crio-5ff6be6375be9b8f29fb475d0a65d9dc397241b7f7402dad97072bdb0d7143f5 WatchSource:0}: Error finding container 5ff6be6375be9b8f29fb475d0a65d9dc397241b7f7402dad97072bdb0d7143f5: Status 404 returned error can't find the container with id 5ff6be6375be9b8f29fb475d0a65d9dc397241b7f7402dad97072bdb0d7143f5 Nov 02 12:07:29.631069 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:29.631035 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:07:29.631188 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:29.631098 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:07:29.631188 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:29.631126 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:07:29.841573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:29.841532 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" event={"ID":"d7f4312e-6933-473d-ace6-29cb4b45512d","Type":"ContainerStarted","Data":"e036f8245438477274dc795f8ddd9dabd9c894d274319656932da9665653c715"} Nov 02 12:07:29.841991 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:29.841606 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" event={"ID":"d7f4312e-6933-473d-ace6-29cb4b45512d","Type":"ContainerStarted","Data":"5ff6be6375be9b8f29fb475d0a65d9dc397241b7f7402dad97072bdb0d7143f5"} Nov 02 12:07:29.963537 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:29.963080 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" podStartSLOduration=1.963062933 podStartE2EDuration="1.963062933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:07:29.961334027 +0000 UTC m=+666.991570281" watchObservedRunningTime="2026-11-02 12:07:29.963062933 +0000 UTC m=+666.993299184" Nov 02 12:07:30.202111 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:30.202071 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:30.204878 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:30.204851 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-th4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 12:07:30.205019 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:30.204899 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 12:07:31.202049 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:31.202012 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-th4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 12:07:31.202573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:31.202102 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 12:07:32.201368 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:32.201338 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-th4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 12:07:32.201643 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:32.201384 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 12:07:33.201960 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:33.201926 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-th4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 12:07:33.202342 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:33.201973 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 12:07:34.202177 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:34.202143 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-th4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 12:07:34.202580 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:34.202194 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 12:07:35.201191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:35.201157 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-th4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 12:07:35.201357 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:35.201206 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 12:07:36.201918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:36.201885 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-th4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 12:07:36.202384 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:36.201931 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 12:07:37.201421 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:37.201389 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-th4ng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 02 12:07:37.201643 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:37.201443 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 02 12:07:38.739550 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:38.739518 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:38.740061 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:38.739905 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:38.740602 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:38.740578 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:07:51.772811 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:51.772778 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:52.091985 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:52.091903 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:53.471190 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:53.471152 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:53.754462 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:07:53.754390 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:07:59.375416 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:07:59.375359 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:08:07.336772 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:07.336736 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:07.582745 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:07.582711 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:25.600109 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:25.600066 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:25.981317 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:25.981281 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:26.289750 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:26.289650 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:26.550071 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:26.549990 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:26.831235 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:26.831135 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:27.084996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:27.084910 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:27.628371 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:27.628339 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:27.929549 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:27.929518 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:29.372999 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:08:29.372962 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:08:30.798689 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:30.798604 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:31.079538 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:31.079440 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:31.418797 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:31.418769 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:31.702160 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:31.702065 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:32.095256 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:32.095181 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:32.346355 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:32.346278 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:34.383956 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:34.383920 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:34.638765 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:34.638687 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:36.920065 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:36.920032 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf"] Nov 02 12:08:36.923631 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:36.923613 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:36.975611 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:36.975587 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-4b9210-serving-cert\"" Nov 02 12:08:36.995629 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:36.995604 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef82d8ca-0100-4d9f-a449-ada84c603c33-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:36.995750 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:36.995641 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgrhm\" (UniqueName: \"kubernetes.io/projected/ef82d8ca-0100-4d9f-a449-ada84c603c33-kube-api-access-sgrhm\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:36.995750 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:36.995671 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:36.995873 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:36.995787 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-metrics-certs\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:37.026662 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.026643 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf"] Nov 02 12:08:37.096338 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.096311 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef82d8ca-0100-4d9f-a449-ada84c603c33-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:37.096457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.096348 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgrhm\" (UniqueName: \"kubernetes.io/projected/ef82d8ca-0100-4d9f-a449-ada84c603c33-kube-api-access-sgrhm\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:37.096457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.096373 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:37.096457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.096419 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-metrics-certs\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:37.096610 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:08:37.096588 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-serving-cert: secret "maas-api-e2e-worker-w3-4b9210-serving-cert" not found Nov 02 12:08:37.096697 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:08:37.096686 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-maas-api-tls podName:ef82d8ca-0100-4d9f-a449-ada84c603c33 nodeName:}" failed. No retries permitted until 2026-11-02 12:08:37.596662185 +0000 UTC m=+734.626898416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-maas-api-tls") pod "maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" (UID: "ef82d8ca-0100-4d9f-a449-ada84c603c33") : secret "maas-api-e2e-worker-w3-4b9210-serving-cert" not found Nov 02 12:08:37.097125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.097099 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef82d8ca-0100-4d9f-a449-ada84c603c33-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:37.098885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.098858 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-metrics-certs\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:37.159056 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.159034 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgrhm\" (UniqueName: \"kubernetes.io/projected/ef82d8ca-0100-4d9f-a449-ada84c603c33-kube-api-access-sgrhm\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:37.220927 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.220866 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:37.466122 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.466078 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:37.600983 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.600922 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:37.603820 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.603794 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-maas-api-tls\") pod \"maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:37.836436 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:37.836404 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:38.211081 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:38.211055 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf"] Nov 02 12:08:38.213726 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:08:38.213702 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef82d8ca_0100_4d9f_a449_ada84c603c33.slice/crio-2afb115e55591d2b1a66e7288521b7ce7834bc25757466fe919d915d676056a8 WatchSource:0}: Error finding container 2afb115e55591d2b1a66e7288521b7ce7834bc25757466fe919d915d676056a8: Status 404 returned error can't find the container with id 2afb115e55591d2b1a66e7288521b7ce7834bc25757466fe919d915d676056a8 Nov 02 12:08:39.162404 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:39.162374 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" event={"ID":"ef82d8ca-0100-4d9f-a449-ada84c603c33","Type":"ContainerStarted","Data":"2afb115e55591d2b1a66e7288521b7ce7834bc25757466fe919d915d676056a8"} Nov 02 12:08:39.287974 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:39.287926 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:39.548826 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:39.548797 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:40.168210 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:40.168177 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" event={"ID":"ef82d8ca-0100-4d9f-a449-ada84c603c33","Type":"ContainerStarted","Data":"61ebbb71b32c9fc611b4ab9770a5792b70fdf9810394260795d84d47dbd5b820"} Nov 02 12:08:40.168418 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:40.168358 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:40.287644 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:40.287597 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" podStartSLOduration=3.390057516 podStartE2EDuration="4.287583619s" totalImagesPullingTime="897.526103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:08:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:08:40.286198427 +0000 UTC m=+737.316434676" watchObservedRunningTime="2026-11-02 12:08:40.287583619 +0000 UTC m=+737.317819900" Nov 02 12:08:41.862786 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:41.862742 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:42.112001 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:42.111945 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:42.359052 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:42.359019 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:42.606161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:42.606125 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:46.178205 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:46.178177 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:08:46.850989 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:46.850955 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942"] Nov 02 12:08:46.854343 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:46.854328 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:46.907922 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:46.907898 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-b1703c-serving-cert\"" Nov 02 12:08:46.923156 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:46.923133 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942"] Nov 02 12:08:46.978127 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:46.978097 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91bc9139-0856-411c-9e4e-f109316b9dbc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:46.978240 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:46.978136 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch2lt\" (UniqueName: \"kubernetes.io/projected/91bc9139-0856-411c-9e4e-f109316b9dbc-kube-api-access-ch2lt\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:46.978308 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:46.978257 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-metrics-certs\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:46.978308 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:46.978299 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:47.079403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.079376 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:47.079559 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.079440 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91bc9139-0856-411c-9e4e-f109316b9dbc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:47.079559 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.079457 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch2lt\" (UniqueName: \"kubernetes.io/projected/91bc9139-0856-411c-9e4e-f109316b9dbc-kube-api-access-ch2lt\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:47.079559 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.079522 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-metrics-certs\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:47.079559 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:08:47.079525 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-serving-cert: secret "maas-api-e2e-worker-w0-b1703c-serving-cert" not found Nov 02 12:08:47.079696 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:08:47.079579 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-maas-api-tls podName:91bc9139-0856-411c-9e4e-f109316b9dbc nodeName:}" failed. No retries permitted until 2026-11-02 12:08:47.579564272 +0000 UTC m=+744.609800504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-maas-api-tls") pod "maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" (UID: "91bc9139-0856-411c-9e4e-f109316b9dbc") : secret "maas-api-e2e-worker-w0-b1703c-serving-cert" not found Nov 02 12:08:47.080066 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.080038 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91bc9139-0856-411c-9e4e-f109316b9dbc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:47.081872 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.081851 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-metrics-certs\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:47.140333 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.140271 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch2lt\" (UniqueName: \"kubernetes.io/projected/91bc9139-0856-411c-9e4e-f109316b9dbc-kube-api-access-ch2lt\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:47.200671 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.200641 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:47.471280 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.471252 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:47.584477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.584439 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:47.586811 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.586792 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1703c-bcd959df-qb942\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:47.767308 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:47.767225 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:48.136982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:48.136957 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942"] Nov 02 12:08:48.139609 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:08:48.139582 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91bc9139_0856_411c_9e4e_f109316b9dbc.slice/crio-0d3f1c367e59a53babdd33d8c0b5f36b4d338e00cf74ebb4717b55ef6c73d686 WatchSource:0}: Error finding container 0d3f1c367e59a53babdd33d8c0b5f36b4d338e00cf74ebb4717b55ef6c73d686: Status 404 returned error can't find the container with id 0d3f1c367e59a53babdd33d8c0b5f36b4d338e00cf74ebb4717b55ef6c73d686 Nov 02 12:08:48.204953 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:48.204927 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" event={"ID":"91bc9139-0856-411c-9e4e-f109316b9dbc","Type":"ContainerStarted","Data":"0d3f1c367e59a53babdd33d8c0b5f36b4d338e00cf74ebb4717b55ef6c73d686"} Nov 02 12:08:49.210780 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:49.210701 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" event={"ID":"91bc9139-0856-411c-9e4e-f109316b9dbc","Type":"ContainerStarted","Data":"87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c"} Nov 02 12:08:49.210780 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:49.210770 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:49.329131 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:49.329082 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" podStartSLOduration=2.552180935 podStartE2EDuration="3.329066875s" totalImagesPullingTime="776.88594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:08:49.32715277 +0000 UTC m=+746.357389020" watchObservedRunningTime="2026-11-02 12:08:49.329066875 +0000 UTC m=+746.359303125" Nov 02 12:08:52.378050 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:52.378012 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:55.222001 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:55.221971 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:08:57.537820 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:08:57.537785 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:08:59.373694 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:08:59.373655 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:09:02.648922 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:02.648888 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:05.957784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:05.957696 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:06.609636 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:06.609602 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:06.910114 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:06.910042 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:07.710733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:07.710695 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:08.066702 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:08.066612 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:09.290901 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:09.290871 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:12.018350 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:12.018319 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:13.232838 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:13.232802 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:13.498347 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:13.498272 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:13.872475 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:13.872396 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:14.195486 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.195452 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:14.434474 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.434431 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2"] Nov 02 12:09:14.439122 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.439102 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.505996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.505918 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.505996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.505981 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.506191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.505998 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.506191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.506030 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.506191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.506143 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.506361 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.506188 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.506361 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.506258 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzld\" (UniqueName: \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-kube-api-access-2kzld\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.506361 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.506291 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.506361 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.506337 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.546685 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.546647 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2"] Nov 02 12:09:14.606913 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.606884 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.606913 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.606914 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607119 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.606933 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607119 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.606980 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607119 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.607011 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607119 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.607046 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzld\" (UniqueName: \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-kube-api-access-2kzld\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607119 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.607068 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607119 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.607113 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607425 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.607200 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607482 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.607434 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607559 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.607512 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607618 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.607598 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607701 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.607686 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.607781 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.607765 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.609380 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.609361 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.609507 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.609400 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.668132 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.668103 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.668372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.668351 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzld\" (UniqueName: \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-kube-api-access-2kzld\") pod \"maas-default-gateway-openshift-default-6dcf757899-gpkx2\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.751565 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.751525 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:14.987686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:14.987645 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:15.173078 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:15.173053 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2"] Nov 02 12:09:15.175211 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:09:15.175187 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bc2fbc_812e_4dd1_9af2_17043cf6adbf.slice/crio-80682a8017c8e3cd941e3b3893ec96ab509c570b62d0c2951118178f0308745d WatchSource:0}: Error finding container 80682a8017c8e3cd941e3b3893ec96ab509c570b62d0c2951118178f0308745d: Status 404 returned error can't find the container with id 80682a8017c8e3cd941e3b3893ec96ab509c570b62d0c2951118178f0308745d Nov 02 12:09:15.177134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:15.177103 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:09:15.177230 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:15.177185 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:09:15.177293 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:15.177228 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:09:15.340429 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:15.340390 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" event={"ID":"56bc2fbc-812e-4dd1-9af2-17043cf6adbf","Type":"ContainerStarted","Data":"3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339"} Nov 02 12:09:15.340595 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:15.340531 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" event={"ID":"56bc2fbc-812e-4dd1-9af2-17043cf6adbf","Type":"ContainerStarted","Data":"80682a8017c8e3cd941e3b3893ec96ab509c570b62d0c2951118178f0308745d"} Nov 02 12:09:15.474198 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:15.474082 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" podStartSLOduration=1.474063133 podStartE2EDuration="1.474063133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:09:15.473134716 +0000 UTC m=+772.503370967" watchObservedRunningTime="2026-11-02 12:09:15.474063133 +0000 UTC m=+772.504299384" Nov 02 12:09:15.765579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:15.757590 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:15.808319 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:15.808284 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:16.077237 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:16.077140 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:16.629185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:16.629151 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:16.768702 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:16.768670 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gpkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 12:09:16.768946 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:16.768920 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 12:09:16.898612 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:16.898547 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:17.752858 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:17.752821 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gpkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 12:09:17.753433 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:17.753397 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 12:09:18.235378 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:18.235341 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:18.496312 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:18.496229 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:18.761590 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:18.756842 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gpkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 12:09:18.761590 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:18.756908 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 12:09:19.072480 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:19.072398 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:19.759926 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:19.759886 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gpkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 12:09:19.760194 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:19.760160 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 12:09:20.765519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:20.759809 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gpkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 12:09:20.765519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:20.759878 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 12:09:21.077606 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:21.077528 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:21.335653 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:21.335588 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:21.673134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:21.673106 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:21.752770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:21.752735 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gpkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 12:09:21.752935 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:21.752795 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 12:09:21.967799 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:21.967725 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:22.751898 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:22.751861 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-gpkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 12:09:22.752055 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:22.751924 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 12:09:24.458750 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:24.458718 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:24.459252 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:24.459009 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:24.459852 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:24.459834 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:09:24.493261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:24.493233 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:26.606667 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:26.606624 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:26.855864 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:26.855833 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:29.371842 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:09:29.371808 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:09:29.613714 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:29.613685 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:34.486867 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.486836 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:34.680385 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.680353 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l"] Nov 02 12:09:34.684905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.684886 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.786659 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.786586 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b1703c\"/\"openshift-service-ca.crt\"" Nov 02 12:09:34.786659 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.786624 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1703c\"/\"simulator-w0-b1703c-kserve-self-signed-certs\"" Nov 02 12:09:34.786659 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.786654 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b1703c\"/\"kube-root-ca.crt\"" Nov 02 12:09:34.786853 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.786823 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1703c\"/\"default-dockercfg-ptjkv\"" Nov 02 12:09:34.788372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.788354 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l"] Nov 02 12:09:34.805471 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.805448 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-home\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.805608 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.805474 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tmp-dir\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.805654 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.805626 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-dshm\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.805710 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.805652 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m4dq\" (UniqueName: \"kubernetes.io/projected/f01b2ae0-f424-46de-976f-fd2188cbfb6d-kube-api-access-6m4dq\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.805762 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.805743 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tls-certs\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.805832 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.805817 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-model-cache\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.906846 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.906808 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-dshm\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.906846 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.906845 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m4dq\" (UniqueName: \"kubernetes.io/projected/f01b2ae0-f424-46de-976f-fd2188cbfb6d-kube-api-access-6m4dq\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.907133 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.906866 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tls-certs\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.907133 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.906930 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-model-cache\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.907133 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.906995 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-home\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.907133 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.907017 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tmp-dir\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.907431 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.907398 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-model-cache\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.907580 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.907446 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tmp-dir\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.907580 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.907482 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-home\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.909215 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.909187 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-dshm\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:34.909547 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:34.909526 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tls-certs\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:35.010697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:35.010673 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m4dq\" (UniqueName: \"kubernetes.io/projected/f01b2ae0-f424-46de-976f-fd2188cbfb6d-kube-api-access-6m4dq\") pod \"simulator-w0-b1703c-kserve-556bf7bff9-ddd4l\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:35.049330 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:35.049263 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:35.295996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:35.295960 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:35.666304 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:35.666281 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l"] Nov 02 12:09:35.668322 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:09:35.668280 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf01b2ae0_f424_46de_976f_fd2188cbfb6d.slice/crio-f5aa216b25241c32ff517ac8cd7aeec22321210094eb8f1c006dc5a07da8e6c0 WatchSource:0}: Error finding container f5aa216b25241c32ff517ac8cd7aeec22321210094eb8f1c006dc5a07da8e6c0: Status 404 returned error can't find the container with id f5aa216b25241c32ff517ac8cd7aeec22321210094eb8f1c006dc5a07da8e6c0 Nov 02 12:09:36.468650 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:36.468610 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" event={"ID":"f01b2ae0-f424-46de-976f-fd2188cbfb6d","Type":"ContainerStarted","Data":"f5aa216b25241c32ff517ac8cd7aeec22321210094eb8f1c006dc5a07da8e6c0"} Nov 02 12:09:38.478655 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:38.478619 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" event={"ID":"f01b2ae0-f424-46de-976f-fd2188cbfb6d","Type":"ContainerStarted","Data":"e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa"} Nov 02 12:09:38.596379 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:38.596332 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" podStartSLOduration=2.7420794649999998 podStartE2EDuration="4.596318525s" totalImagesPullingTime="1.85423906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:09:38.594864275 +0000 UTC m=+795.625100525" watchObservedRunningTime="2026-11-02 12:09:38.596318525 +0000 UTC m=+795.626554774" Nov 02 12:09:39.296331 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:39.296288 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:39.340975 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:39.340937 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:39.587861 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:39.587727 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:39.840054 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:39.839986 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:44.496036 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:44.495984 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:45.296348 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:45.296314 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:45.308989 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:45.308960 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:45.518363 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:45.518335 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:09:46.601735 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.601705 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k"] Nov 02 12:09:46.605221 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.605199 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.705818 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.705790 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-dshm\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.705979 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.705828 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tls-certs\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.705979 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.705859 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-home\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.706104 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.705971 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-model-cache\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.706104 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.706034 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8wp\" (UniqueName: \"kubernetes.io/projected/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-kube-api-access-lv8wp\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.706213 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.706101 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tmp-dir\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.709330 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.709304 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4b9210\"/\"premium-w3-4b9210-kserve-self-signed-certs\"" Nov 02 12:09:46.709455 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.709333 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4b9210\"/\"openshift-service-ca.crt\"" Nov 02 12:09:46.709455 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.709372 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4b9210\"/\"default-dockercfg-6254k\"" Nov 02 12:09:46.709455 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.709376 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4b9210\"/\"kube-root-ca.crt\"" Nov 02 12:09:46.712075 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.712056 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k"] Nov 02 12:09:46.807172 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.807137 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-dshm\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.807353 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.807191 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tls-certs\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.807353 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.807214 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-home\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.807353 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.807261 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-model-cache\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.807353 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.807320 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8wp\" (UniqueName: \"kubernetes.io/projected/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-kube-api-access-lv8wp\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.807353 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.807339 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tmp-dir\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.807757 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.807705 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tmp-dir\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.807872 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.807771 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-model-cache\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.807982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.807955 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-home\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.809745 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.809719 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-dshm\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.811414 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.811390 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tls-certs\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.835682 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.835634 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:46.868386 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.868336 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8wp\" (UniqueName: \"kubernetes.io/projected/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-kube-api-access-lv8wp\") pod \"premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:46.917037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:46.917012 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:47.291160 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:47.291133 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k"] Nov 02 12:09:47.293708 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:09:47.293683 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec93f568_75f8_4ff3_8718_1c3e86fe5b49.slice/crio-d9503d6feb998ea90b8f9f68477b30dcc109df0a7829bb93c8e014d3b1bbfbca WatchSource:0}: Error finding container d9503d6feb998ea90b8f9f68477b30dcc109df0a7829bb93c8e014d3b1bbfbca: Status 404 returned error can't find the container with id d9503d6feb998ea90b8f9f68477b30dcc109df0a7829bb93c8e014d3b1bbfbca Nov 02 12:09:47.517865 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:47.517831 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" event={"ID":"ec93f568-75f8-4ff3-8718-1c3e86fe5b49","Type":"ContainerStarted","Data":"b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b"} Nov 02 12:09:47.517865 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:47.517876 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" event={"ID":"ec93f568-75f8-4ff3-8718-1c3e86fe5b49","Type":"ContainerStarted","Data":"d9503d6feb998ea90b8f9f68477b30dcc109df0a7829bb93c8e014d3b1bbfbca"} Nov 02 12:09:47.633808 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:47.633720 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podStartSLOduration=1.633705438 podStartE2EDuration="1.633705438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:09:47.633165617 +0000 UTC m=+804.663401867" watchObservedRunningTime="2026-11-02 12:09:47.633705438 +0000 UTC m=+804.663941688" Nov 02 12:09:47.917266 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:47.917236 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:49.337915 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:49.337881 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:49.615671 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:49.615559 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:49.863846 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:49.863811 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:54.621073 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:54.621031 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:09:56.917669 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:56.917633 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:56.929843 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:56.929822 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:57.578328 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:57.578294 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:09:59.374268 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:09:59.374231 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:09:59.506559 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:09:59.506520 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:04.533266 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:04.533232 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:08.049741 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:08.049705 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:09.494426 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:09.494392 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:12.066540 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.066492 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d"] Nov 02 12:10:12.072696 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.072666 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.124995 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.124970 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4b9210-openshift-default-dockercfg-fqcg6\"" Nov 02 12:10:12.176868 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.176842 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d"] Nov 02 12:10:12.219625 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.219599 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.219745 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.219648 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.219745 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.219670 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.219822 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.219743 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.219822 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.219793 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.219822 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.219817 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.219917 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.219837 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.219948 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.219931 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.219991 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.219979 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnk2h\" (UniqueName: \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-kube-api-access-wnk2h\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321165 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321085 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnk2h\" (UniqueName: \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-kube-api-access-wnk2h\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321165 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321125 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321358 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321168 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321358 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321192 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321358 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321225 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321358 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321251 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321358 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321286 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321358 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321318 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321358 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321346 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321774 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321629 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321774 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321682 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321876 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321805 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.321950 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.321929 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.322198 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.322151 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.323716 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.323697 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.324121 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.324102 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.382044 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.382017 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.382189 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.382112 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnk2h\" (UniqueName: \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-kube-api-access-wnk2h\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.385982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.385846 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:12.451265 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.451082 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:12.769659 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.769626 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d"] Nov 02 12:10:12.771857 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:10:12.771810 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5aacb6_f61a_4d84_957e_96b9241c40e2.slice/crio-989f2158f50c6150803c1bbe7bf717b0919c2229b2d2b4757926890fbf3d06fe WatchSource:0}: Error finding container 989f2158f50c6150803c1bbe7bf717b0919c2229b2d2b4757926890fbf3d06fe: Status 404 returned error can't find the container with id 989f2158f50c6150803c1bbe7bf717b0919c2229b2d2b4757926890fbf3d06fe Nov 02 12:10:12.774173 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.774136 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:10:12.774271 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.774222 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:10:12.774271 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:12.774260 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:10:13.638296 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:13.638264 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" event={"ID":"0d5aacb6-f61a-4d84-957e-96b9241c40e2","Type":"ContainerStarted","Data":"5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a"} Nov 02 12:10:13.638825 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:13.638321 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" event={"ID":"0d5aacb6-f61a-4d84-957e-96b9241c40e2","Type":"ContainerStarted","Data":"989f2158f50c6150803c1bbe7bf717b0919c2229b2d2b4757926890fbf3d06fe"} Nov 02 12:10:13.808458 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:13.808394 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" podStartSLOduration=1.808380406 podStartE2EDuration="1.808380406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:10:13.808347287 +0000 UTC m=+830.838583551" watchObservedRunningTime="2026-11-02 12:10:13.808380406 +0000 UTC m=+830.838616659" Nov 02 12:10:14.386440 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:14.386402 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:14.387792 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:14.387767 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 12:10:14.387903 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:14.387809 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" podUID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 12:10:14.507536 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:14.507479 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:15.387290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:15.387252 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 12:10:15.387736 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:15.387313 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" podUID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 12:10:16.386907 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:16.386873 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 12:10:16.387165 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:16.386926 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" podUID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 12:10:17.387103 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:17.387066 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 12:10:17.387480 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:17.387121 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" podUID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 12:10:18.386972 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:18.386934 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 12:10:18.387132 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:18.386989 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" podUID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 12:10:19.039248 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:19.039207 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:19.608076 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:19.608047 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:19.650535 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:19.650488 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:19.651398 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:19.651366 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:19.652691 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:19.652671 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:10:24.496347 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:24.496316 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:25.811386 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:25.811345 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:26.418619 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:26.418564 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:26.995562 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:26.995526 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:27.398628 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:27.398545 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:29.371695 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:10:29.371661 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:10:29.706317 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:29.706274 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:31.707603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:31.707564 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:31.963771 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:31.963687 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:33.346949 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:33.346909 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:33.607113 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:33.607026 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:34.906602 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:34.906566 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:35.152371 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:35.152339 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:39.965695 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:39.965617 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:41.489134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.489095 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v"] Nov 02 12:10:41.492677 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.492652 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.594286 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.594258 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-b6f3d8\"/\"default-dockercfg-rc8n6\"" Nov 02 12:10:41.594445 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.594297 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-b6f3d8\"/\"praxis-route-b6f3d8-kserve-self-signed-certs\"" Nov 02 12:10:41.594445 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.594310 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-b6f3d8\"/\"openshift-service-ca.crt\"" Nov 02 12:10:41.594445 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.594441 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-b6f3d8\"/\"kube-root-ca.crt\"" Nov 02 12:10:41.596170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.596151 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v"] Nov 02 12:10:41.613845 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.613823 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tmp-dir\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.613931 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.613851 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9hw\" (UniqueName: \"kubernetes.io/projected/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-kube-api-access-kl9hw\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.613931 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.613884 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-model-cache\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.614034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.614016 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tls-certs\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.614076 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.614044 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-home\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.614159 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.614146 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-dshm\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.691904 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.691870 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:41.715139 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.715106 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-dshm\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.715257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.715160 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tmp-dir\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.715257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.715192 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9hw\" (UniqueName: \"kubernetes.io/projected/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-kube-api-access-kl9hw\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.715257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.715230 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-model-cache\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.715393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.715278 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tls-certs\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.715453 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.715423 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-home\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.715659 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.715638 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tmp-dir\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.715772 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.715753 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-home\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.715999 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.715983 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-model-cache\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.717533 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.717483 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-dshm\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.717782 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.717766 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tls-certs\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.781198 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.781136 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9hw\" (UniqueName: \"kubernetes.io/projected/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-kube-api-access-kl9hw\") pod \"praxis-route-b6f3d8-kserve-6bff944b87-srk9v\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:41.804812 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:41.804791 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:42.187399 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:42.187371 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v"] Nov 02 12:10:42.190126 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:10:42.190100 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c378df9_5e93_47c3_8e29_dbbf0e291f4b.slice/crio-ef3c702ed9e0a68ba15de7e402ac01f3032f8d35d42a6a2f8c61ec996db8abb1 WatchSource:0}: Error finding container ef3c702ed9e0a68ba15de7e402ac01f3032f8d35d42a6a2f8c61ec996db8abb1: Status 404 returned error can't find the container with id ef3c702ed9e0a68ba15de7e402ac01f3032f8d35d42a6a2f8c61ec996db8abb1 Nov 02 12:10:42.792944 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:42.792912 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" event={"ID":"2c378df9-5e93-47c3-8e29-dbbf0e291f4b","Type":"ContainerStarted","Data":"1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874"} Nov 02 12:10:42.793393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:42.792950 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" event={"ID":"2c378df9-5e93-47c3-8e29-dbbf0e291f4b","Type":"ContainerStarted","Data":"ef3c702ed9e0a68ba15de7e402ac01f3032f8d35d42a6a2f8c61ec996db8abb1"} Nov 02 12:10:42.805285 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:42.805262 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:42.912257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:42.912195 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" podStartSLOduration=1.912182331 podStartE2EDuration="1.912182331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:10:42.910703552 +0000 UTC m=+859.940939804" watchObservedRunningTime="2026-11-02 12:10:42.912182331 +0000 UTC m=+859.942418580" Nov 02 12:10:44.813735 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:44.813704 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:49.878296 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:49.878260 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:51.805877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:51.805843 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:51.818442 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:51.818418 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:51.853863 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:51.853841 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:10:52.997815 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:52.997780 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:53.684286 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:53.684248 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:54.064649 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:54.064564 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:54.806210 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:54.806174 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:58.034875 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:58.034841 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:58.324592 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:10:58.324520 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:10:59.372370 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:10:59.372334 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:11:00.001832 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:00.001799 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:01.294448 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:01.294414 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:01.542702 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:01.542667 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:04.949622 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:04.949585 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:05.195972 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:05.195942 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:06.783422 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.783386 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7"] Nov 02 12:11:06.786954 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.786939 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.833723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.833695 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-model-cache\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.833875 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.833730 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tls-certs\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.833875 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.833802 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-home\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.833996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.833885 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tmp-dir\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.833996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.833939 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn2qj\" (UniqueName: \"kubernetes.io/projected/0713e0a4-bf82-4e18-b9ba-8a428c04816e-kube-api-access-gn2qj\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.834068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.834016 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-dshm\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.841342 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.841322 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4b9210\"/\"e2e-conflict-a-68877c04-kserve-self-signed-certs\"" Nov 02 12:11:06.892929 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.892905 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7"] Nov 02 12:11:06.935186 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.935147 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tmp-dir\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.935381 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.935211 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2qj\" (UniqueName: \"kubernetes.io/projected/0713e0a4-bf82-4e18-b9ba-8a428c04816e-kube-api-access-gn2qj\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.935381 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.935265 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-dshm\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.935381 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.935360 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-model-cache\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.935656 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.935387 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tls-certs\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.935656 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.935426 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-home\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.935891 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.935866 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-home\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.936386 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.936302 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-model-cache\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.936747 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.936726 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tmp-dir\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.947050 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.938789 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-dshm\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.947050 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.939326 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tls-certs\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:06.968346 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.968317 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:06.998450 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:06.998425 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2qj\" (UniqueName: \"kubernetes.io/projected/0713e0a4-bf82-4e18-b9ba-8a428c04816e-kube-api-access-gn2qj\") pod \"e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:07.098756 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:07.098685 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:07.483785 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:07.483763 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7"] Nov 02 12:11:07.486027 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:11:07.486000 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0713e0a4_bf82_4e18_b9ba_8a428c04816e.slice/crio-5552deefcdc6adc0225886def684c9cace2b8788b5c4ce1132cdd4a0d9829c1e WatchSource:0}: Error finding container 5552deefcdc6adc0225886def684c9cace2b8788b5c4ce1132cdd4a0d9829c1e: Status 404 returned error can't find the container with id 5552deefcdc6adc0225886def684c9cace2b8788b5c4ce1132cdd4a0d9829c1e Nov 02 12:11:07.921202 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:07.921174 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" event={"ID":"0713e0a4-bf82-4e18-b9ba-8a428c04816e","Type":"ContainerStarted","Data":"c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd"} Nov 02 12:11:07.921617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:07.921211 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" event={"ID":"0713e0a4-bf82-4e18-b9ba-8a428c04816e","Type":"ContainerStarted","Data":"5552deefcdc6adc0225886def684c9cace2b8788b5c4ce1132cdd4a0d9829c1e"} Nov 02 12:11:08.040225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:08.040181 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" podStartSLOduration=2.040167567 podStartE2EDuration="2.040167567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:11:08.038872233 +0000 UTC m=+885.069108483" watchObservedRunningTime="2026-11-02 12:11:08.040167567 +0000 UTC m=+885.070403816" Nov 02 12:11:08.099534 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:08.099487 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:08.394328 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:08.394242 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:10.332338 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:10.332311 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:15.316252 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:15.316219 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:17.099282 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:17.099254 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:17.111430 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:17.111408 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:17.982207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:17.982179 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:11:20.475514 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:20.475458 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:25.434255 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:25.434220 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:27.709351 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.709319 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd"] Nov 02 12:11:27.713205 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.713189 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.811085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.811055 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd"] Nov 02 12:11:27.824523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.823585 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj4gh\" (UniqueName: \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-kube-api-access-hj4gh\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.824523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.823688 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.824523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.823740 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.824523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.823774 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.824523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.823832 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.824523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.823868 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab605a2c-6923-4a4c-8423-fe54a21caac0-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.824523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.823914 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.824523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.823970 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.824523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.824024 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925180 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925154 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925206 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj4gh\" (UniqueName: \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-kube-api-access-hj4gh\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925252 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925283 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925418 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925309 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925418 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925353 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925418 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925378 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab605a2c-6923-4a4c-8423-fe54a21caac0-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925418 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925411 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925662 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925463 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925767 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925744 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925833 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925784 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.925985 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925955 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.926089 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.925970 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.926089 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.926056 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab605a2c-6923-4a4c-8423-fe54a21caac0-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.927743 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.927722 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.927979 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.927958 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.986007 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.985944 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:27.986289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:27.986267 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj4gh\" (UniqueName: \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-kube-api-access-hj4gh\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:28.025046 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:28.025025 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:28.447036 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:28.447010 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd"] Nov 02 12:11:28.449203 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:11:28.449175 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab605a2c_6923_4a4c_8423_fe54a21caac0.slice/crio-09f0b0a8b917884aa13f1b924fff099d1b623575ba0387f38dcfda05be45da58 WatchSource:0}: Error finding container 09f0b0a8b917884aa13f1b924fff099d1b623575ba0387f38dcfda05be45da58: Status 404 returned error can't find the container with id 09f0b0a8b917884aa13f1b924fff099d1b623575ba0387f38dcfda05be45da58 Nov 02 12:11:28.451588 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:28.451549 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:11:28.451693 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:28.451623 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:11:28.451693 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:28.451655 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:11:29.023922 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:29.023885 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" event={"ID":"ab605a2c-6923-4a4c-8423-fe54a21caac0","Type":"ContainerStarted","Data":"99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab"} Nov 02 12:11:29.024406 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:29.023940 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" event={"ID":"ab605a2c-6923-4a4c-8423-fe54a21caac0","Type":"ContainerStarted","Data":"09f0b0a8b917884aa13f1b924fff099d1b623575ba0387f38dcfda05be45da58"} Nov 02 12:11:29.025239 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:29.025223 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:29.026842 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:29.026818 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 12:11:29.026956 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:29.026867 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 12:11:29.168517 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:29.167766 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" podStartSLOduration=2.16774512 podStartE2EDuration="2.16774512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:11:29.164131516 +0000 UTC m=+906.194367767" watchObservedRunningTime="2026-11-02 12:11:29.16774512 +0000 UTC m=+906.197981372" Nov 02 12:11:29.393263 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:11:29.393187 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:11:30.025712 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:30.025675 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 12:11:30.026094 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:30.025744 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 12:11:30.492652 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:30.492617 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:31.025762 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:31.025728 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 12:11:31.026142 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:31.025781 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 12:11:32.025809 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:32.025774 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 12:11:32.026184 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:32.025825 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 12:11:33.025892 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:33.025857 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 12:11:33.026289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:33.025909 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 12:11:33.937137 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:33.937107 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp"] Nov 02 12:11:33.940708 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:33.940692 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.026207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.026162 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 12:11:34.026608 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.026234 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 12:11:34.043395 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.043364 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4b9210\"/\"e2e-conflict-b-68877c04-kserve-self-signed-certs\"" Nov 02 12:11:34.045593 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.045570 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp"] Nov 02 12:11:34.104307 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.104263 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-model-cache\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.104307 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.104307 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnf8b\" (UniqueName: \"kubernetes.io/projected/6c57127b-c333-4b45-b7f7-cf759cfdc04b-kube-api-access-hnf8b\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.104603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.104372 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-dshm\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.104603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.104387 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tls-certs\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.104603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.104406 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-home\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.104603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.104429 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tmp-dir\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.144585 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.144545 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:34.205782 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.205702 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-dshm\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.205782 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.205739 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tls-certs\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.205959 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.205853 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-home\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.205959 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.205895 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tmp-dir\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.206027 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.205995 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-model-cache\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.206063 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.206046 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnf8b\" (UniqueName: \"kubernetes.io/projected/6c57127b-c333-4b45-b7f7-cf759cfdc04b-kube-api-access-hnf8b\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.206296 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.206273 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-model-cache\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.206478 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.206456 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-home\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.206627 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.206492 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tmp-dir\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.208587 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.208546 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tls-certs\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.208587 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.208581 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-dshm\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.277204 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.277147 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnf8b\" (UniqueName: \"kubernetes.io/projected/6c57127b-c333-4b45-b7f7-cf759cfdc04b-kube-api-access-hnf8b\") pod \"e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.282848 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.282823 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:34.672142 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:34.672115 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp"] Nov 02 12:11:34.673477 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:11:34.673455 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c57127b_c333_4b45_b7f7_cf759cfdc04b.slice/crio-87fc6bf1ad8b590e5ce5243dbe9f47e2c28f72ff48abb68dc0a921b088a94a6f WatchSource:0}: Error finding container 87fc6bf1ad8b590e5ce5243dbe9f47e2c28f72ff48abb68dc0a921b088a94a6f: Status 404 returned error can't find the container with id 87fc6bf1ad8b590e5ce5243dbe9f47e2c28f72ff48abb68dc0a921b088a94a6f Nov 02 12:11:35.025782 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:35.025704 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 02 12:11:35.025782 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:35.025754 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 02 12:11:35.055045 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:35.055006 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" event={"ID":"6c57127b-c333-4b45-b7f7-cf759cfdc04b","Type":"ContainerStarted","Data":"300e3da055af531fc97c1bfac8498ff010157fe4936f0ab93f2f6d902cc1a972"} Nov 02 12:11:35.055472 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:35.055062 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" event={"ID":"6c57127b-c333-4b45-b7f7-cf759cfdc04b","Type":"ContainerStarted","Data":"87fc6bf1ad8b590e5ce5243dbe9f47e2c28f72ff48abb68dc0a921b088a94a6f"} Nov 02 12:11:35.173897 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:35.173836 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" podStartSLOduration=2.17381876 podStartE2EDuration="2.17381876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:11:35.172040905 +0000 UTC m=+912.202277156" watchObservedRunningTime="2026-11-02 12:11:35.17381876 +0000 UTC m=+912.204055013" Nov 02 12:11:35.283754 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:35.283687 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:35.348660 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:35.348621 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:35.609001 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:35.608912 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:35.861001 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:35.860926 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:36.056001 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:36.055969 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:36.056478 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:36.056422 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:36.057403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:36.057384 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:11:40.550632 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:40.550601 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:41.020200 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:41.020163 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:41.276515 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:41.276414 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:44.283603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:44.283565 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:44.295933 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:44.295910 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:45.117850 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:45.117822 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:11:45.447416 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:45.447375 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:50.756812 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:50.756771 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:53.225868 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:53.225831 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:56.112910 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:56.112873 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:11:58.223881 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.223850 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6"] Nov 02 12:11:58.227246 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.227229 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.279459 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.279435 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b1703c-openshift-default-dockercfg-rb2rb\"" Nov 02 12:11:58.330949 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.330924 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6"] Nov 02 12:11:58.338363 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.338337 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.338483 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.338385 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.338483 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.338411 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.338579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.338534 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.338614 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.338599 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.338715 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.338680 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.338762 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.338746 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp24x\" (UniqueName: \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-kube-api-access-lp24x\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.338813 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.338799 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.338869 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.338855 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.439791 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.439761 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.439942 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.439803 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.439942 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.439857 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.439942 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.439924 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.440115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.439960 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.440115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.439999 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp24x\" (UniqueName: \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-kube-api-access-lp24x\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.440115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.440046 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.440115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.440073 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.440315 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.440112 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.440371 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.440345 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.440436 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.440345 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.440518 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.440435 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.440596 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.440552 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.440654 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.440610 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.442485 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.442461 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.442671 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.442652 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.500372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.500313 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.500518 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.500482 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp24x\" (UniqueName: \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-kube-api-access-lp24x\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.540431 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.540403 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:11:58.968251 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.968102 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6"] Nov 02 12:11:58.971051 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:11:58.971025 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a53a8c4_121c_4da4_be4b_7d21a697d1f8.slice/crio-849d1d5581b70d92036112fc3e69b46b19d23b54122987bd5d13552e2c349544 WatchSource:0}: Error finding container 849d1d5581b70d92036112fc3e69b46b19d23b54122987bd5d13552e2c349544: Status 404 returned error can't find the container with id 849d1d5581b70d92036112fc3e69b46b19d23b54122987bd5d13552e2c349544 Nov 02 12:11:58.973574 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.973533 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:11:58.973695 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.973616 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:11:58.973695 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:58.973656 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:11:59.178774 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:59.178740 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" event={"ID":"8a53a8c4-121c-4da4-be4b-7d21a697d1f8","Type":"ContainerStarted","Data":"3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d"} Nov 02 12:11:59.178941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:59.178789 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" event={"ID":"8a53a8c4-121c-4da4-be4b-7d21a697d1f8","Type":"ContainerStarted","Data":"849d1d5581b70d92036112fc3e69b46b19d23b54122987bd5d13552e2c349544"} Nov 02 12:11:59.302584 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:59.301960 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" podStartSLOduration=1.3019459420000001 podStartE2EDuration="1.301945942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:11:59.301560448 +0000 UTC m=+936.331796698" watchObservedRunningTime="2026-11-02 12:11:59.301945942 +0000 UTC m=+936.332182191" Nov 02 12:11:59.382870 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:11:59.382837 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:11:59.541309 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:11:59.541276 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:12:00.542455 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:00.542418 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 12:12:00.542864 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:00.542472 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 12:12:01.179536 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:01.177062 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:01.541455 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:01.541373 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 12:12:01.541455 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:01.541422 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 12:12:02.540861 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:02.540830 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 12:12:02.541227 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:02.540884 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 12:12:03.541316 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:03.541277 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 12:12:03.541744 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:03.541327 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 12:12:04.541112 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:04.541080 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 12:12:04.541276 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:04.541131 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 12:12:05.541415 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:05.541379 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 02 12:12:05.541813 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:05.541431 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 02 12:12:05.848974 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:05.848864 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:06.200728 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:06.200702 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:06.908262 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:06.908233 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:12:06.908765 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:06.908588 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:12:06.909326 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:06.909305 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:12:08.738721 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:08.738670 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:09.568794 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:09.568757 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:10.706594 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:10.706565 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v"] Nov 02 12:12:10.883130 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:10.883097 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:11.187068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:11.187031 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:16.206714 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:16.206678 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:21.242250 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:21.242219 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:24.530255 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:24.530221 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:24.969237 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:24.969200 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:25.481353 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.481318 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw"] Nov 02 12:12:25.484821 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.484797 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.546187 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.546150 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-036f88-openshift-default-dockercfg-gfh5b\"" Nov 02 12:12:25.597849 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.597821 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw"] Nov 02 12:12:25.604137 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.604104 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-credential-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.604298 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.604158 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-podinfo\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.604298 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.604282 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-envoy\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.604424 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.604377 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-token\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.604533 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.604515 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnx6c\" (UniqueName: \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-kube-api-access-mnx6c\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.604604 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.604556 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-data\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.604662 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.604637 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-certs\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.604710 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.604675 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.604806 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.604781 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8106eef-1121-4fe1-aba8-99b6f485bcee-istiod-ca-cert\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.679037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.679005 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:25.706018 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.705980 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnx6c\" (UniqueName: \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-kube-api-access-mnx6c\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.706225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.706040 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-data\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.706225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.706086 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-certs\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.706225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.706123 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.706225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.706167 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8106eef-1121-4fe1-aba8-99b6f485bcee-istiod-ca-cert\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.706464 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.706236 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-credential-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.706464 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.706270 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-podinfo\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.706464 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.706314 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-envoy\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.706464 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.706364 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-token\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.729673 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.707355 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-data\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.729673 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.707660 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-certs\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.729673 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.707893 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.729673 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.709571 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-credential-socket\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.729673 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.729012 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-podinfo\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.729673 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.729243 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-envoy\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.729673 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.729674 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8106eef-1121-4fe1-aba8-99b6f485bcee-istiod-ca-cert\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.737378 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.737222 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" podUID="2c378df9-5e93-47c3-8e29-dbbf0e291f4b" containerName="main" containerID="cri-o://1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874" gracePeriod=45 Nov 02 12:12:25.770573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.769196 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnx6c\" (UniqueName: \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-kube-api-access-mnx6c\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.770573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.769975 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-token\") pod \"e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.798901 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.798877 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:25.843839 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:25.843688 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" podUID="2c378df9-5e93-47c3-8e29-dbbf0e291f4b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused" Nov 02 12:12:26.040734 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.040706 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:12:26.110984 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.110945 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-home\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-home\") pod \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " Nov 02 12:12:26.111145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.111006 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tmp-dir\") pod \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " Nov 02 12:12:26.111145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.111117 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-kube-api-access-kl9hw\" (UniqueName: \"kubernetes.io/projected/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-kube-api-access-kl9hw\") pod \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " Nov 02 12:12:26.111258 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.111174 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-model-cache\") pod \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " Nov 02 12:12:26.111258 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.111228 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-dshm\") pod \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " Nov 02 12:12:26.111357 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.111262 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tls-certs\") pod \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\" (UID: \"2c378df9-5e93-47c3-8e29-dbbf0e291f4b\") " Nov 02 12:12:26.111792 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.111723 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-home" pod "2c378df9-5e93-47c3-8e29-dbbf0e291f4b" (UID: "2c378df9-5e93-47c3-8e29-dbbf0e291f4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:26.111792 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.111740 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-model-cache" pod "2c378df9-5e93-47c3-8e29-dbbf0e291f4b" (UID: "2c378df9-5e93-47c3-8e29-dbbf0e291f4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:26.112009 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.111986 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tmp-dir" pod "2c378df9-5e93-47c3-8e29-dbbf0e291f4b" (UID: "2c378df9-5e93-47c3-8e29-dbbf0e291f4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:26.114219 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.114182 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tls-certs" pod "2c378df9-5e93-47c3-8e29-dbbf0e291f4b" (UID: "2c378df9-5e93-47c3-8e29-dbbf0e291f4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:12:26.114219 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.114195 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-kube-api-access-kl9hw" pod "2c378df9-5e93-47c3-8e29-dbbf0e291f4b" (UID: "2c378df9-5e93-47c3-8e29-dbbf0e291f4b"). InnerVolumeSpecName "kube-api-access-kl9hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:12:26.114437 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.114418 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-dshm" pod "2c378df9-5e93-47c3-8e29-dbbf0e291f4b" (UID: "2c378df9-5e93-47c3-8e29-dbbf0e291f4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:26.215297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.215166 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:26.215297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.215211 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:26.215297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.215221 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl9hw\" (UniqueName: \"kubernetes.io/projected/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-kube-api-access-kl9hw\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:26.215297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.215230 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:26.215297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.215239 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:26.215297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.215247 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c378df9-5e93-47c3-8e29-dbbf0e291f4b-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:26.235133 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.235106 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw"] Nov 02 12:12:26.236238 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:12:26.236198 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8106eef_1121_4fe1_aba8_99b6f485bcee.slice/crio-9bafbec8f79a0b9da456e84951a4c2f5ff64d4c41d343afb2accd3bddc1b9a62 WatchSource:0}: Error finding container 9bafbec8f79a0b9da456e84951a4c2f5ff64d4c41d343afb2accd3bddc1b9a62: Status 404 returned error can't find the container with id 9bafbec8f79a0b9da456e84951a4c2f5ff64d4c41d343afb2accd3bddc1b9a62 Nov 02 12:12:26.238926 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.238893 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:12:26.239046 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.238956 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:12:26.239046 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.238983 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:12:26.318370 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.318342 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" event={"ID":"e8106eef-1121-4fe1-aba8-99b6f485bcee","Type":"ContainerStarted","Data":"9bafbec8f79a0b9da456e84951a4c2f5ff64d4c41d343afb2accd3bddc1b9a62"} Nov 02 12:12:26.319892 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.319863 4312 generic.go:356] "Generic (PLEG): container finished" podID="2c378df9-5e93-47c3-8e29-dbbf0e291f4b" containerID="1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874" exitCode=0 Nov 02 12:12:26.320029 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.319903 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" event={"ID":"2c378df9-5e93-47c3-8e29-dbbf0e291f4b","Type":"ContainerDied","Data":"1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874"} Nov 02 12:12:26.320029 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.319911 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" Nov 02 12:12:26.320029 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.319931 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v" event={"ID":"2c378df9-5e93-47c3-8e29-dbbf0e291f4b","Type":"ContainerDied","Data":"ef3c702ed9e0a68ba15de7e402ac01f3032f8d35d42a6a2f8c61ec996db8abb1"} Nov 02 12:12:26.323565 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.319952 4312 scope.go:122] "RemoveContainer" containerID="1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874" Nov 02 12:12:26.333696 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.333673 4312 scope.go:122] "RemoveContainer" containerID="1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874" Nov 02 12:12:26.333967 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:12:26.333947 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874\": container with ID starting with 1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874 not found: ID does not exist" containerID="1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874" Nov 02 12:12:26.334032 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.333977 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874"} err="failed to get container status \"1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874\": rpc error: code = NotFound desc = could not find container \"1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874\": container with ID starting with 1c60c80fdac3a2223c046142e2f40442ead97c6ee886bcdb80b6712e97c98874 not found: ID does not exist" Nov 02 12:12:26.335172 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.335126 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:26.484435 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.484405 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v"] Nov 02 12:12:26.488352 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.488326 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-b6f3d8/praxis-route-b6f3d8-kserve-6bff944b87-srk9v"] Nov 02 12:12:26.618049 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:26.617960 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:27.075283 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:27.075253 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c378df9-5e93-47c3-8e29-dbbf0e291f4b" path="/var/lib/kubelet/pods/2c378df9-5e93-47c3-8e29-dbbf0e291f4b/volumes" Nov 02 12:12:27.327017 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:27.326946 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" event={"ID":"e8106eef-1121-4fe1-aba8-99b6f485bcee","Type":"ContainerStarted","Data":"c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b"} Nov 02 12:12:27.499864 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:27.499813 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" podStartSLOduration=2.499798233 podStartE2EDuration="2.499798233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:12:27.497146977 +0000 UTC m=+964.527383229" watchObservedRunningTime="2026-11-02 12:12:27.499798233 +0000 UTC m=+964.530034483" Nov 02 12:12:27.799430 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:27.799395 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:27.804314 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:27.804289 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:28.332349 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:28.332319 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:28.333078 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:28.333057 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:12:29.234845 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:29.234800 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp"] Nov 02 12:12:29.360570 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:29.360538 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:29.374524 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:12:29.374391 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:12:30.608889 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:30.608858 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:30.985732 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:30.985702 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:31.775183 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:31.775153 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:32.605959 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:32.605929 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7"] Nov 02 12:12:32.733811 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:32.733780 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:34.216247 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:34.216200 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:34.468020 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:34.467942 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:37.104261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:37.104230 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:41.998747 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:41.998713 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:44.254248 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.254189 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" podUID="6c57127b-c333-4b45-b7f7-cf759cfdc04b" containerName="main" containerID="cri-o://300e3da055af531fc97c1bfac8498ff010157fe4936f0ab93f2f6d902cc1a972" gracePeriod=45 Nov 02 12:12:44.411399 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.411367 4312 generic.go:356] "Generic (PLEG): container finished" podID="6c57127b-c333-4b45-b7f7-cf759cfdc04b" containerID="300e3da055af531fc97c1bfac8498ff010157fe4936f0ab93f2f6d902cc1a972" exitCode=0 Nov 02 12:12:44.411586 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.411434 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" event={"ID":"6c57127b-c333-4b45-b7f7-cf759cfdc04b","Type":"ContainerDied","Data":"300e3da055af531fc97c1bfac8498ff010157fe4936f0ab93f2f6d902cc1a972"} Nov 02 12:12:44.500750 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.500727 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:12:44.592426 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.592377 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c57127b-c333-4b45-b7f7-cf759cfdc04b-kube-api-access-hnf8b\" (UniqueName: \"kubernetes.io/projected/6c57127b-c333-4b45-b7f7-cf759cfdc04b-kube-api-access-hnf8b\") pod \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " Nov 02 12:12:44.592546 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.592430 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-model-cache\") pod \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " Nov 02 12:12:44.592546 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.592541 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tls-certs\") pod \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " Nov 02 12:12:44.592675 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.592611 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tmp-dir\") pod \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " Nov 02 12:12:44.592733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.592706 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-dshm\") pod \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " Nov 02 12:12:44.592791 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.592734 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-home\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-home\") pod \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\" (UID: \"6c57127b-c333-4b45-b7f7-cf759cfdc04b\") " Nov 02 12:12:44.592791 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.592750 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-model-cache" pod "6c57127b-c333-4b45-b7f7-cf759cfdc04b" (UID: "6c57127b-c333-4b45-b7f7-cf759cfdc04b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:44.592928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.592878 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tmp-dir" pod "6c57127b-c333-4b45-b7f7-cf759cfdc04b" (UID: "6c57127b-c333-4b45-b7f7-cf759cfdc04b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:44.592986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.592960 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-home" pod "6c57127b-c333-4b45-b7f7-cf759cfdc04b" (UID: "6c57127b-c333-4b45-b7f7-cf759cfdc04b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:44.593178 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.593161 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:44.593242 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.593184 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:44.593242 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.593198 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:44.594676 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.594644 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tls-certs" pod "6c57127b-c333-4b45-b7f7-cf759cfdc04b" (UID: "6c57127b-c333-4b45-b7f7-cf759cfdc04b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:12:44.594768 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.594699 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-dshm" pod "6c57127b-c333-4b45-b7f7-cf759cfdc04b" (UID: "6c57127b-c333-4b45-b7f7-cf759cfdc04b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:44.594768 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.594705 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c57127b-c333-4b45-b7f7-cf759cfdc04b-kube-api-access-hnf8b" pod "6c57127b-c333-4b45-b7f7-cf759cfdc04b" (UID: "6c57127b-c333-4b45-b7f7-cf759cfdc04b"). InnerVolumeSpecName "kube-api-access-hnf8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:12:44.694474 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.694450 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c57127b-c333-4b45-b7f7-cf759cfdc04b-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:44.694474 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.694474 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnf8b\" (UniqueName: \"kubernetes.io/projected/6c57127b-c333-4b45-b7f7-cf759cfdc04b-kube-api-access-hnf8b\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:44.694660 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:44.694484 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c57127b-c333-4b45-b7f7-cf759cfdc04b-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:45.417191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:45.417167 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" Nov 02 12:12:45.417622 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:45.417190 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp" event={"ID":"6c57127b-c333-4b45-b7f7-cf759cfdc04b","Type":"ContainerDied","Data":"87fc6bf1ad8b590e5ce5243dbe9f47e2c28f72ff48abb68dc0a921b088a94a6f"} Nov 02 12:12:45.417622 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:45.417232 4312 scope.go:122] "RemoveContainer" containerID="300e3da055af531fc97c1bfac8498ff010157fe4936f0ab93f2f6d902cc1a972" Nov 02 12:12:45.581353 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:45.581330 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp"] Nov 02 12:12:45.631220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:45.631197 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-b-68877c04-kserve-8745bc4b6-l2sdp"] Nov 02 12:12:46.923483 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:46.923449 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:47.075057 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.075027 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c57127b-c333-4b45-b7f7-cf759cfdc04b" path="/var/lib/kubelet/pods/6c57127b-c333-4b45-b7f7-cf759cfdc04b/volumes" Nov 02 12:12:47.625384 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.625338 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" podUID="0713e0a4-bf82-4e18-b9ba-8a428c04816e" containerName="main" containerID="cri-o://c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd" gracePeriod=45 Nov 02 12:12:47.780718 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.780682 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:47.895625 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.895602 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:12:47.925820 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.925798 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0713e0a4-bf82-4e18-b9ba-8a428c04816e-kube-api-access-gn2qj\" (UniqueName: \"kubernetes.io/projected/0713e0a4-bf82-4e18-b9ba-8a428c04816e-kube-api-access-gn2qj\") pod \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " Nov 02 12:12:47.926185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.925860 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tls-certs\") pod \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " Nov 02 12:12:47.926185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.925881 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-dshm\") pod \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " Nov 02 12:12:47.926185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.925930 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-home\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-home\") pod \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " Nov 02 12:12:47.926185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.925950 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-model-cache\") pod \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " Nov 02 12:12:47.926185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.925994 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tmp-dir\") pod \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\" (UID: \"0713e0a4-bf82-4e18-b9ba-8a428c04816e\") " Nov 02 12:12:47.926467 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.926192 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-home" pod "0713e0a4-bf82-4e18-b9ba-8a428c04816e" (UID: "0713e0a4-bf82-4e18-b9ba-8a428c04816e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:47.926467 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.926236 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-model-cache" pod "0713e0a4-bf82-4e18-b9ba-8a428c04816e" (UID: "0713e0a4-bf82-4e18-b9ba-8a428c04816e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:47.926467 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.926306 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tmp-dir" pod "0713e0a4-bf82-4e18-b9ba-8a428c04816e" (UID: "0713e0a4-bf82-4e18-b9ba-8a428c04816e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:47.926467 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.926456 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:47.926726 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.926476 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:47.926726 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.926491 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:47.927849 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.927827 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0713e0a4-bf82-4e18-b9ba-8a428c04816e-kube-api-access-gn2qj" pod "0713e0a4-bf82-4e18-b9ba-8a428c04816e" (UID: "0713e0a4-bf82-4e18-b9ba-8a428c04816e"). InnerVolumeSpecName "kube-api-access-gn2qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:12:47.927931 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.927866 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-dshm" pod "0713e0a4-bf82-4e18-b9ba-8a428c04816e" (UID: "0713e0a4-bf82-4e18-b9ba-8a428c04816e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:12:47.928017 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:47.927988 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tls-certs" pod "0713e0a4-bf82-4e18-b9ba-8a428c04816e" (UID: "0713e0a4-bf82-4e18-b9ba-8a428c04816e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:12:48.027018 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.026996 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn2qj\" (UniqueName: \"kubernetes.io/projected/0713e0a4-bf82-4e18-b9ba-8a428c04816e-kube-api-access-gn2qj\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:48.027018 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.027017 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0713e0a4-bf82-4e18-b9ba-8a428c04816e-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:48.027148 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.027026 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0713e0a4-bf82-4e18-b9ba-8a428c04816e-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:12:48.439826 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.439793 4312 generic.go:356] "Generic (PLEG): container finished" podID="0713e0a4-bf82-4e18-b9ba-8a428c04816e" containerID="c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd" exitCode=0 Nov 02 12:12:48.439989 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.439844 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" Nov 02 12:12:48.439989 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.439864 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" event={"ID":"0713e0a4-bf82-4e18-b9ba-8a428c04816e","Type":"ContainerDied","Data":"c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd"} Nov 02 12:12:48.439989 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.439904 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7" event={"ID":"0713e0a4-bf82-4e18-b9ba-8a428c04816e","Type":"ContainerDied","Data":"5552deefcdc6adc0225886def684c9cace2b8788b5c4ce1132cdd4a0d9829c1e"} Nov 02 12:12:48.439989 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.439925 4312 scope.go:122] "RemoveContainer" containerID="c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd" Nov 02 12:12:48.449184 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.449167 4312 scope.go:122] "RemoveContainer" containerID="c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd" Nov 02 12:12:48.449422 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:12:48.449402 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd\": container with ID starting with c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd not found: ID does not exist" containerID="c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd" Nov 02 12:12:48.449488 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.449440 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd"} err="failed to get container status \"c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd\": rpc error: code = NotFound desc = could not find container \"c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd\": container with ID starting with c898407b17bdeecc12ad3547b8538305194972ebd3816dd1c55f6c3eb00b52dd not found: ID does not exist" Nov 02 12:12:48.604637 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.604615 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7"] Nov 02 12:12:48.654220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:48.654196 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/e2e-conflict-a-68877c04-kserve-5b6c888c6d-4wqm7"] Nov 02 12:12:49.075262 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:49.075227 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0713e0a4-bf82-4e18-b9ba-8a428c04816e" path="/var/lib/kubelet/pods/0713e0a4-bf82-4e18-b9ba-8a428c04816e/volumes" Nov 02 12:12:49.097253 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:49.097227 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:49.877623 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:49.877582 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:50.120930 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:50.120895 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:52.157914 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:52.157879 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:52.409822 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:52.409736 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:55.551686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:55.551649 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:55.889988 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:55.889907 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:57.197911 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:57.197876 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:12:58.393834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.393802 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf"] Nov 02 12:12:58.394249 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.394184 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0713e0a4-bf82-4e18-b9ba-8a428c04816e" containerName="main" Nov 02 12:12:58.394249 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.394195 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0713e0a4-bf82-4e18-b9ba-8a428c04816e" containerName="main" Nov 02 12:12:58.394249 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.394207 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2c378df9-5e93-47c3-8e29-dbbf0e291f4b" containerName="main" Nov 02 12:12:58.394249 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.394211 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c378df9-5e93-47c3-8e29-dbbf0e291f4b" containerName="main" Nov 02 12:12:58.394249 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.394229 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6c57127b-c333-4b45-b7f7-cf759cfdc04b" containerName="main" Nov 02 12:12:58.394249 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.394234 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c57127b-c333-4b45-b7f7-cf759cfdc04b" containerName="main" Nov 02 12:12:58.394435 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.394293 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c57127b-c333-4b45-b7f7-cf759cfdc04b" containerName="main" Nov 02 12:12:58.394435 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.394303 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0713e0a4-bf82-4e18-b9ba-8a428c04816e" containerName="main" Nov 02 12:12:58.394435 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.394310 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c378df9-5e93-47c3-8e29-dbbf0e291f4b" containerName="main" Nov 02 12:12:58.398899 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.398879 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.443420 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.443399 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9"] Nov 02 12:12:58.446934 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.446918 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.504138 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.504110 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf"] Nov 02 12:12:58.529589 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.529559 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.529730 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.529601 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.529730 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.529722 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.529848 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.529752 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.529848 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.529781 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5txhz\" (UniqueName: \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-kube-api-access-5txhz\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.529952 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.529866 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3270a9-c28f-4705-afde-b7272562b52c-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.529952 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.529912 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjz9\" (UniqueName: \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-kube-api-access-ndjz9\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.530057 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.529959 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.530057 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.529983 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.530161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.530054 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.530161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.530076 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.530161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.530091 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.530161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.530109 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.530360 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.530170 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.530360 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.530212 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.530360 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.530247 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.530360 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.530265 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3270a9-c28f-4705-afde-b7272562b52c-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.530360 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.530338 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.557123 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.557091 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9"] Nov 02 12:12:58.630872 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.630844 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.630986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.630879 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.630986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.630895 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.630986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.630930 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.631127 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631106 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.631185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631150 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txhz\" (UniqueName: \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-kube-api-access-5txhz\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.631238 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631191 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3270a9-c28f-4705-afde-b7272562b52c-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.631290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631247 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjz9\" (UniqueName: \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-kube-api-access-ndjz9\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.631290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631253 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.631399 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631305 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.631399 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631348 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.631399 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631379 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.631596 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631406 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.631596 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631433 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.631596 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631459 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.631596 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631523 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.631596 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631563 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.631596 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631595 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.631904 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631615 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3270a9-c28f-4705-afde-b7272562b52c-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.631904 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631672 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.631904 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631707 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.631904 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631873 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.632107 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.631945 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.632107 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.632030 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.632216 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.632118 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.632216 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.632130 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3270a9-c28f-4705-afde-b7272562b52c-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.632308 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.632259 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.632355 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.632337 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.633656 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.633636 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.634222 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.634193 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.634850 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.634679 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3270a9-c28f-4705-afde-b7272562b52c-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.634850 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.634712 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.694731 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.694707 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.694855 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.694838 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.695006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.694982 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjz9\" (UniqueName: \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-kube-api-access-ndjz9\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.695069 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.695037 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txhz\" (UniqueName: \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-kube-api-access-5txhz\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:58.711376 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.711359 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:58.758803 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:58.758777 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:12:59.138220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.138197 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf"] Nov 02 12:12:59.139936 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:12:59.139914 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dff9c5_8b62_4a98_a162_1a6fa3c1f4b4.slice/crio-f4482681d36440d38319030c49c63272f9de98679ef3f5c4c96ac07d92830c83 WatchSource:0}: Error finding container f4482681d36440d38319030c49c63272f9de98679ef3f5c4c96ac07d92830c83: Status 404 returned error can't find the container with id f4482681d36440d38319030c49c63272f9de98679ef3f5c4c96ac07d92830c83 Nov 02 12:12:59.142216 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.142186 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:12:59.142292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.142251 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:12:59.142292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.142281 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:12:59.192961 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.192932 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9"] Nov 02 12:12:59.194087 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:12:59.194064 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3270a9_c28f_4705_afde_b7272562b52c.slice/crio-01b2124e9784f575d0cfff1259ae3374cf98555ef72e2660deb56a890ba7973c WatchSource:0}: Error finding container 01b2124e9784f575d0cfff1259ae3374cf98555ef72e2660deb56a890ba7973c: Status 404 returned error can't find the container with id 01b2124e9784f575d0cfff1259ae3374cf98555ef72e2660deb56a890ba7973c Nov 02 12:12:59.196334 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.196308 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:12:59.196412 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.196371 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:12:59.196412 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.196400 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:12:59.377527 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:12:59.376340 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:12:59.494234 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.494199 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" event={"ID":"ba3270a9-c28f-4705-afde-b7272562b52c","Type":"ContainerStarted","Data":"8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8"} Nov 02 12:12:59.494722 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.494262 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" event={"ID":"ba3270a9-c28f-4705-afde-b7272562b52c","Type":"ContainerStarted","Data":"01b2124e9784f575d0cfff1259ae3374cf98555ef72e2660deb56a890ba7973c"} Nov 02 12:12:59.496003 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.495970 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" event={"ID":"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4","Type":"ContainerStarted","Data":"720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5"} Nov 02 12:12:59.496140 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.496013 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" event={"ID":"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4","Type":"ContainerStarted","Data":"f4482681d36440d38319030c49c63272f9de98679ef3f5c4c96ac07d92830c83"} Nov 02 12:12:59.616254 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.616185 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" podStartSLOduration=1.6161640369999999 podStartE2EDuration="1.616164037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:12:59.615362637 +0000 UTC m=+996.645598889" watchObservedRunningTime="2026-11-02 12:12:59.616164037 +0000 UTC m=+996.646400290" Nov 02 12:12:59.712917 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.712808 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:12:59.739520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.739355 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" podStartSLOduration=1.739338084 podStartE2EDuration="1.739338084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:12:59.737436913 +0000 UTC m=+996.767673158" watchObservedRunningTime="2026-11-02 12:12:59.739338084 +0000 UTC m=+996.769574333" Nov 02 12:12:59.759738 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:12:59.759595 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:13:00.713329 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:00.713300 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 12:13:00.713905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:00.713346 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 12:13:00.777521 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:00.776582 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 12:13:00.777521 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:00.776637 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 12:13:01.712733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:01.712694 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 12:13:01.712966 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:01.712943 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 12:13:01.773988 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:01.773533 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:01.773988 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:01.773896 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 12:13:01.773988 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:01.773935 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 12:13:02.090037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:02.089940 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:02.712159 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:02.712127 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 12:13:02.712344 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:02.712177 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 12:13:02.759836 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:02.759803 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 12:13:02.760085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:02.760057 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 12:13:03.712701 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:03.712665 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 12:13:03.713221 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:03.712726 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 12:13:03.759876 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:03.759845 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 12:13:03.760100 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:03.760075 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 12:13:04.713519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:04.712792 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 12:13:04.713519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:04.712838 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 12:13:04.763518 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:04.763302 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 12:13:04.763518 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:04.763352 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 12:13:05.630094 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.630060 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg"] Nov 02 12:13:05.633871 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.633852 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.644339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.644318 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-dshm\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.644452 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.644347 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-home\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.644452 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.644378 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tmp-dir\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.644604 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.644465 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m44q9\" (UniqueName: \"kubernetes.io/projected/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-kube-api-access-m44q9\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.644604 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.644489 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tls-certs\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.644604 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.644555 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-model-cache\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.712839 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.712809 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Nov 02 12:13:05.713033 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.713007 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Nov 02 12:13:05.735627 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.735598 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4b9210\"/\"distinct-w3-4b9210-kserve-self-signed-certs\"" Nov 02 12:13:05.739370 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.739342 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg"] Nov 02 12:13:05.745819 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.745785 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-dshm\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.745929 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.745832 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-home\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.745929 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.745880 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tmp-dir\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.746059 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.745929 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m44q9\" (UniqueName: \"kubernetes.io/projected/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-kube-api-access-m44q9\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.746059 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.745966 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tls-certs\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.746059 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.746001 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-model-cache\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.746456 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.746434 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-model-cache\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.746662 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.746635 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tmp-dir\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.747096 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.747063 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-home\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.748744 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.748697 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-dshm\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.749608 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.749479 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tls-certs\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.759603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.759581 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 12:13:05.759780 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.759752 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 12:13:05.819867 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.819005 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m44q9\" (UniqueName: \"kubernetes.io/projected/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-kube-api-access-m44q9\") pod \"distinct-w3-4b9210-kserve-6c849bffdb-jvwtg\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:05.830414 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.830380 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:05.948412 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:05.948375 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:06.438453 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.438432 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg"] Nov 02 12:13:06.440578 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:13:06.440549 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7db5c59e_e7cc_4bb5_97c8_dfb5fc8a99ab.slice/crio-4fd99b5a1b52f552a45ed25f6b04bf4c66520a736bee86c3a8a351cf89738010 WatchSource:0}: Error finding container 4fd99b5a1b52f552a45ed25f6b04bf4c66520a736bee86c3a8a351cf89738010: Status 404 returned error can't find the container with id 4fd99b5a1b52f552a45ed25f6b04bf4c66520a736bee86c3a8a351cf89738010 Nov 02 12:13:06.533959 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.533925 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" event={"ID":"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab","Type":"ContainerStarted","Data":"bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295"} Nov 02 12:13:06.534093 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.533971 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" event={"ID":"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab","Type":"ContainerStarted","Data":"4fd99b5a1b52f552a45ed25f6b04bf4c66520a736bee86c3a8a351cf89738010"} Nov 02 12:13:06.657748 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.657699 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podStartSLOduration=1.6576852199999998 podStartE2EDuration="1.65768522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:13:06.656397898 +0000 UTC m=+1003.686634149" watchObservedRunningTime="2026-11-02 12:13:06.65768522 +0000 UTC m=+1003.687921470" Nov 02 12:13:06.717012 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.716956 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:13:06.717237 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.717221 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:13:06.717935 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.717919 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:13:06.763721 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.763703 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:13:06.764036 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.763963 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:13:06.764666 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.764651 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:13:06.949917 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.949888 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:06.961663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:06.961614 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:07.209139 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:07.209109 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:07.457996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:07.457957 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:09.096694 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:09.096661 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:12.257425 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:12.257388 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:14.614937 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.614907 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9"] Nov 02 12:13:14.618306 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.618285 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.673573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.673547 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d52846-b9e0-4b0c-b03e-f2e90b668969-tls-certs\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.673687 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.673651 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-dshm\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.673687 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.673679 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmqf\" (UniqueName: \"kubernetes.io/projected/67d52846-b9e0-4b0c-b03e-f2e90b668969-kube-api-access-5lmqf\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.673758 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.673709 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-tmp-dir\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.673758 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.673746 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-model-cache\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.673824 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.673802 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-home\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.721402 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.720938 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-03c4e8\"/\"default-dockercfg-7bxqj\"" Nov 02 12:13:14.721402 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.721015 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-03c4e8\"/\"openshift-service-ca.crt\"" Nov 02 12:13:14.721402 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.721098 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-03c4e8\"/\"auth-test-model-03c4e8-kserve-self-signed-certs\"" Nov 02 12:13:14.728092 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.721763 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-03c4e8\"/\"kube-root-ca.crt\"" Nov 02 12:13:14.731252 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.731225 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9"] Nov 02 12:13:14.771554 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.771523 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:14.774606 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.774575 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-model-cache\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.774606 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.774609 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-home\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.774793 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.774668 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d52846-b9e0-4b0c-b03e-f2e90b668969-tls-certs\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.774793 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.774738 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-dshm\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.774793 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.774763 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmqf\" (UniqueName: \"kubernetes.io/projected/67d52846-b9e0-4b0c-b03e-f2e90b668969-kube-api-access-5lmqf\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.774793 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.774787 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-tmp-dir\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.775086 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.775063 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-model-cache\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.775163 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.775129 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-home\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.775203 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.775185 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-tmp-dir\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.777113 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.777090 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-dshm\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.777406 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.777387 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d52846-b9e0-4b0c-b03e-f2e90b668969-tls-certs\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.835393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.835371 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmqf\" (UniqueName: \"kubernetes.io/projected/67d52846-b9e0-4b0c-b03e-f2e90b668969-kube-api-access-5lmqf\") pod \"auth-test-model-03c4e8-kserve-7c4dc959df-2rch9\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:14.930446 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:14.930417 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:15.303934 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:15.303910 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9"] Nov 02 12:13:15.306088 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:13:15.306060 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67d52846_b9e0_4b0c_b03e_f2e90b668969.slice/crio-b4b05311d5151e811b3fac2250b84bd9160005143b323c8d6e5a7b739de00283 WatchSource:0}: Error finding container b4b05311d5151e811b3fac2250b84bd9160005143b323c8d6e5a7b739de00283: Status 404 returned error can't find the container with id b4b05311d5151e811b3fac2250b84bd9160005143b323c8d6e5a7b739de00283 Nov 02 12:13:15.597446 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:15.597365 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" event={"ID":"67d52846-b9e0-4b0c-b03e-f2e90b668969","Type":"ContainerStarted","Data":"f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b"} Nov 02 12:13:15.597446 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:15.597407 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" event={"ID":"67d52846-b9e0-4b0c-b03e-f2e90b668969","Type":"ContainerStarted","Data":"b4b05311d5151e811b3fac2250b84bd9160005143b323c8d6e5a7b739de00283"} Nov 02 12:13:15.715908 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:15.715831 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" podStartSLOduration=1.715813019 podStartE2EDuration="1.715813019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:13:15.715174741 +0000 UTC m=+1012.745410992" watchObservedRunningTime="2026-11-02 12:13:15.715813019 +0000 UTC m=+1012.746049271" Nov 02 12:13:15.931568 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:15.931529 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:15.949686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:15.949659 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:15.962062 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:15.962043 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:16.613579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:16.613550 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:13:17.310403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:17.310366 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:21.209865 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:21.209831 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:21.462514 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:21.462425 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:22.187565 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:22.187529 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:24.931060 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:24.931031 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:24.943123 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:24.943097 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:25.665737 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:25.665707 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:13:27.188157 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:27.188127 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:29.385677 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:13:29.385623 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:13:32.198810 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:32.198776 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:37.182986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:37.182903 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:42.489293 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:42.489252 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:47.393511 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:47.393462 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:48.108352 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:48.108322 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:52.419300 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:52.419262 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:57.129984 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:57.129950 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:57.500659 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:57.500625 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:13:59.165769 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.165727 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp"] Nov 02 12:13:59.169461 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.169434 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.252928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.252895 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.253091 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.252953 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.253091 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.253029 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.253091 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.253071 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.253261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.253130 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.253261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.253182 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.253261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.253209 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.253550 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.253271 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.253550 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.253312 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddp5f\" (UniqueName: \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-kube-api-access-ddp5f\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.281685 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.281652 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp"] Nov 02 12:13:59.354934 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.354894 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.354934 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.354927 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.355177 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.354949 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.355177 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.355080 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.355177 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.355149 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.355350 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.355186 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.355350 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.355224 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.355350 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.355251 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddp5f\" (UniqueName: \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-kube-api-access-ddp5f\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.355530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.355366 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.355530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.355381 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.355530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.355382 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.355708 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.355689 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.355769 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.355757 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.356217 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.356189 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.357395 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.357371 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.358327 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.358299 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.375248 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:13:59.375213 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:13:59.375634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.375613 4312 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 12:13:59.375727 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.375656 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 12:13:59.416293 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.416234 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.416293 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.416271 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddp5f\" (UniqueName: \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-kube-api-access-ddp5f\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.483075 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.483055 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:13:59.915221 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.913804 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp"] Nov 02 12:13:59.917637 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:13:59.917610 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cc0b38_8f11_4c28_835f_c5bbe213ee19.slice/crio-38f90cc0c56ee137f84f3f3c3cf007915695554dfd6b93adbf2ac6784dbbefed WatchSource:0}: Error finding container 38f90cc0c56ee137f84f3f3c3cf007915695554dfd6b93adbf2ac6784dbbefed: Status 404 returned error can't find the container with id 38f90cc0c56ee137f84f3f3c3cf007915695554dfd6b93adbf2ac6784dbbefed Nov 02 12:13:59.920767 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.920725 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:13:59.920855 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.920814 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:13:59.920900 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:13:59.920857 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:14:00.880591 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:00.880394 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" event={"ID":"d5cc0b38-8f11-4c28-835f-c5bbe213ee19","Type":"ContainerStarted","Data":"a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667"} Nov 02 12:14:00.880591 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:00.880464 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" event={"ID":"d5cc0b38-8f11-4c28-835f-c5bbe213ee19","Type":"ContainerStarted","Data":"38f90cc0c56ee137f84f3f3c3cf007915695554dfd6b93adbf2ac6784dbbefed"} Nov 02 12:14:01.068520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:01.060864 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" podStartSLOduration=2.060843732 podStartE2EDuration="2.060843732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:14:01.060091286 +0000 UTC m=+1058.090327539" watchObservedRunningTime="2026-11-02 12:14:01.060843732 +0000 UTC m=+1058.091079984" Nov 02 12:14:01.489274 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:01.487106 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:14:01.489653 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:01.489611 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 12:14:01.489876 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:01.489850 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" podUID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 12:14:02.484170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:02.483725 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 12:14:02.484170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:02.483784 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" podUID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 12:14:02.497817 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:02.497790 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:02.906854 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:02.906777 4312 generic.go:356] "Generic (PLEG): container finished" podID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerID="c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b" exitCode=0 Nov 02 12:14:02.907005 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:02.906852 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" event={"ID":"e8106eef-1121-4fe1-aba8-99b6f485bcee","Type":"ContainerDied","Data":"c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b"} Nov 02 12:14:02.907254 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:02.907238 4312 scope.go:122] "RemoveContainer" containerID="c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b" Nov 02 12:14:02.908465 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:02.908437 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:14:02.908590 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:02.908550 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:14:02.908655 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:02.908604 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:14:03.484385 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:03.484352 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 12:14:03.484796 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:03.484401 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" podUID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 12:14:03.913969 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:03.913937 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" event={"ID":"e8106eef-1121-4fe1-aba8-99b6f485bcee","Type":"ContainerStarted","Data":"bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab"} Nov 02 12:14:04.487119 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:04.487077 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 12:14:04.487753 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:04.487719 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" podUID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 12:14:04.799877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:04.799783 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:14:04.800054 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:04.799958 4312 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 12:14:04.800054 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:04.800015 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 12:14:05.488884 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:05.488845 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 12:14:05.489483 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:05.489450 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" podUID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 12:14:05.800123 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:05.800016 4312 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 12:14:05.800123 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:05.800078 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 12:14:06.488522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:06.488159 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 12:14:06.488522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:06.488233 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" podUID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 12:14:06.799681 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:06.799595 4312 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 12:14:06.800139 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:06.800114 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 12:14:07.458376 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:07.458339 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:07.720281 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:07.720198 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:14:07.721058 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:07.721032 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:14:07.721949 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:07.721925 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:14:07.799517 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:07.799450 4312 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 12:14:07.799644 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:07.799546 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 12:14:08.799638 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:08.799604 4312 patch_prober.go:29] interesting pod/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Nov 02 12:14:08.800034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:08.799661 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Nov 02 12:14:09.445097 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:09.445058 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:09.803747 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:09.803659 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:14:09.804202 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:09.803995 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:14:09.804697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:09.804678 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:14:12.403614 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:12.403579 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:16.656562 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:16.645592 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:18.443174 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:18.443144 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:18.695850 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:18.695777 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:19.063707 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:19.063631 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:20.276954 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:20.276925 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:20.522900 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:20.522864 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:23.709245 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:23.709218 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:24.820681 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:24.820649 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:25.233230 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:25.233193 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:28.597109 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:28.597077 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:29.372120 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:14:29.372085 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:14:30.912792 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:30.912757 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:32.603304 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:32.603269 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:32.895081 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:32.894999 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:33.675394 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:33.675360 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:36.765397 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:36.765365 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:38.086525 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:38.086479 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:38.433160 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:38.433132 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:38.693684 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:38.693560 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:39.865987 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:39.865955 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:42.582521 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:42.579969 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:42.853141 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:42.853014 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:44.215591 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:44.215553 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:44.744519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:44.744117 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:45.029723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:45.029636 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:46.584217 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:46.584188 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:47.129178 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:47.129145 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2"] Nov 02 12:14:47.129409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:47.129335 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerName="istio-proxy" containerID="cri-o://3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339" gracePeriod=30 Nov 02 12:14:47.346169 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:47.346124 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:47.753411 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:47.753380 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:50.524786 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:50.524749 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:52.123516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.123455 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:52.375812 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.375754 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:52.385779 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.385759 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:14:52.472612 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.472589 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-certs\") pod \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " Nov 02 12:14:52.472753 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.472648 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-kube-api-access-2kzld\" (UniqueName: \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-kube-api-access-2kzld\") pod \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " Nov 02 12:14:52.472753 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.472692 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-data\") pod \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " Nov 02 12:14:52.472866 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.472782 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-token\" (UniqueName: \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-token\") pod \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " Nov 02 12:14:52.472866 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.472829 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-credential-socket\") pod \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " Nov 02 12:14:52.472866 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.472831 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-certs" pod "56bc2fbc-812e-4dd1-9af2-17043cf6adbf" (UID: "56bc2fbc-812e-4dd1-9af2-17043cf6adbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:14:52.473024 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.472980 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-envoy\") pod \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " Nov 02 12:14:52.473082 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473033 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-data" pod "56bc2fbc-812e-4dd1-9af2-17043cf6adbf" (UID: "56bc2fbc-812e-4dd1-9af2-17043cf6adbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:14:52.473138 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473079 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-credential-socket" pod "56bc2fbc-812e-4dd1-9af2-17043cf6adbf" (UID: "56bc2fbc-812e-4dd1-9af2-17043cf6adbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:14:52.473138 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473083 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istiod-ca-cert\") pod \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " Nov 02 12:14:52.473239 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473148 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-podinfo\") pod \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " Nov 02 12:14:52.473239 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473189 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-socket\") pod \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\" (UID: \"56bc2fbc-812e-4dd1-9af2-17043cf6adbf\") " Nov 02 12:14:52.473392 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473369 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istiod-ca-cert" pod "56bc2fbc-812e-4dd1-9af2-17043cf6adbf" (UID: "56bc2fbc-812e-4dd1-9af2-17043cf6adbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:14:52.473526 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473481 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-socket" pod "56bc2fbc-812e-4dd1-9af2-17043cf6adbf" (UID: "56bc2fbc-812e-4dd1-9af2-17043cf6adbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:14:52.473700 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473683 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:14:52.473770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473702 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:14:52.473770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473711 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:14:52.473770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473719 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:14:52.473770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.473728 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:14:52.475351 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.475325 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-podinfo" pod "56bc2fbc-812e-4dd1-9af2-17043cf6adbf" (UID: "56bc2fbc-812e-4dd1-9af2-17043cf6adbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:14:52.475438 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.475375 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-token" pod "56bc2fbc-812e-4dd1-9af2-17043cf6adbf" (UID: "56bc2fbc-812e-4dd1-9af2-17043cf6adbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:14:52.475486 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.475439 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-envoy" pod "56bc2fbc-812e-4dd1-9af2-17043cf6adbf" (UID: "56bc2fbc-812e-4dd1-9af2-17043cf6adbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:14:52.475486 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.475447 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-kube-api-access-2kzld" pod "56bc2fbc-812e-4dd1-9af2-17043cf6adbf" (UID: "56bc2fbc-812e-4dd1-9af2-17043cf6adbf"). InnerVolumeSpecName "kube-api-access-2kzld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:14:52.575231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.575203 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:14:52.575341 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.575237 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:14:52.575341 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.575253 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kzld\" (UniqueName: \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-kube-api-access-2kzld\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:14:52.575341 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:52.575268 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56bc2fbc-812e-4dd1-9af2-17043cf6adbf-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:14:53.185207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:53.185173 4312 generic.go:356] "Generic (PLEG): container finished" podID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerID="3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339" exitCode=0 Nov 02 12:14:53.185657 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:53.185237 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" Nov 02 12:14:53.185657 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:53.185264 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" event={"ID":"56bc2fbc-812e-4dd1-9af2-17043cf6adbf","Type":"ContainerDied","Data":"3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339"} Nov 02 12:14:53.185657 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:53.185323 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2" event={"ID":"56bc2fbc-812e-4dd1-9af2-17043cf6adbf","Type":"ContainerDied","Data":"80682a8017c8e3cd941e3b3893ec96ab509c570b62d0c2951118178f0308745d"} Nov 02 12:14:53.185657 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:53.185347 4312 scope.go:122] "RemoveContainer" containerID="3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339" Nov 02 12:14:53.196075 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:53.196058 4312 scope.go:122] "RemoveContainer" containerID="3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339" Nov 02 12:14:53.196339 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:14:53.196319 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339\": container with ID starting with 3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339 not found: ID does not exist" containerID="3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339" Nov 02 12:14:53.196423 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:53.196347 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339"} err="failed to get container status \"3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339\": rpc error: code = NotFound desc = could not find container \"3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339\": container with ID starting with 3e76d037caf2a5664da5af40e1c456bd649148fc609d4b3af49dd8d0bb467339 not found: ID does not exist" Nov 02 12:14:53.399184 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:53.399104 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2"] Nov 02 12:14:53.448069 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:53.448044 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-gpkx2"] Nov 02 12:14:55.075468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:55.075433 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" path="/var/lib/kubelet/pods/56bc2fbc-812e-4dd1-9af2-17043cf6adbf/volumes" Nov 02 12:14:55.894241 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:55.894204 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:57.058525 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:57.051018 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:57.384880 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:14:57.384752 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:14:59.374594 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:14:59.374552 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:15:00.187692 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.187662 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq"] Nov 02 12:15:00.188100 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.188086 4312 cpu_manager.go:414] "containerMap: removing container" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerName="istio-proxy" Nov 02 12:15:00.188146 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.188101 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerName="istio-proxy" Nov 02 12:15:00.188187 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.188178 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="56bc2fbc-812e-4dd1-9af2-17043cf6adbf" containerName="istio-proxy" Nov 02 12:15:00.190850 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.190832 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" Nov 02 12:15:00.242954 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.242927 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\"" Nov 02 12:15:00.271922 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.271897 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqw5p\" (UniqueName: \"kubernetes.io/projected/4d2029df-8360-4cb6-925b-5622b6a3178d-kube-api-access-xqw5p\") pod \"maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq\" (UID: \"4d2029df-8360-4cb6-925b-5622b6a3178d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" Nov 02 12:15:00.272037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.271954 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2029df-8360-4cb6-925b-5622b6a3178d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq\" (UID: \"4d2029df-8360-4cb6-925b-5622b6a3178d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" Nov 02 12:15:00.295842 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.295815 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq"] Nov 02 12:15:00.372635 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.372606 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2029df-8360-4cb6-925b-5622b6a3178d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq\" (UID: \"4d2029df-8360-4cb6-925b-5622b6a3178d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" Nov 02 12:15:00.372764 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.372709 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqw5p\" (UniqueName: \"kubernetes.io/projected/4d2029df-8360-4cb6-925b-5622b6a3178d-kube-api-access-xqw5p\") pod \"maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq\" (UID: \"4d2029df-8360-4cb6-925b-5622b6a3178d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" Nov 02 12:15:00.373251 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.373232 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2029df-8360-4cb6-925b-5622b6a3178d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq\" (UID: \"4d2029df-8360-4cb6-925b-5622b6a3178d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" Nov 02 12:15:00.433818 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.433788 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqw5p\" (UniqueName: \"kubernetes.io/projected/4d2029df-8360-4cb6-925b-5622b6a3178d-kube-api-access-xqw5p\") pod \"maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq\" (UID: \"4d2029df-8360-4cb6-925b-5622b6a3178d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" Nov 02 12:15:00.509140 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.509083 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" Nov 02 12:15:00.878339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.878317 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq"] Nov 02 12:15:00.880438 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:15:00.880410 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2029df_8360_4cb6_925b_5622b6a3178d.slice/crio-29836896509f5e367b349784c0387769e4a3401a25b2a62e97edf0c3afc95d77 WatchSource:0}: Error finding container 29836896509f5e367b349784c0387769e4a3401a25b2a62e97edf0c3afc95d77: Status 404 returned error can't find the container with id 29836896509f5e367b349784c0387769e4a3401a25b2a62e97edf0c3afc95d77 Nov 02 12:15:00.882257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:00.882242 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 12:15:01.108568 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:01.108539 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:01.442322 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:01.442285 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" event={"ID":"4d2029df-8360-4cb6-925b-5622b6a3178d","Type":"ContainerStarted","Data":"29836896509f5e367b349784c0387769e4a3401a25b2a62e97edf0c3afc95d77"} Nov 02 12:15:05.464231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:05.464197 4312 generic.go:356] "Generic (PLEG): container finished" podID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerID="37d8ed181939a9428f7b1413aaa20b73475ed4eacca4787e079f0ed7d117e36a" exitCode=60 Nov 02 12:15:05.464662 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:05.464241 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" event={"ID":"4d2029df-8360-4cb6-925b-5622b6a3178d","Type":"ContainerDied","Data":"37d8ed181939a9428f7b1413aaa20b73475ed4eacca4787e079f0ed7d117e36a"} Nov 02 12:15:05.464662 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:05.464464 4312 scope.go:122] "RemoveContainer" containerID="37d8ed181939a9428f7b1413aaa20b73475ed4eacca4787e079f0ed7d117e36a" Nov 02 12:15:06.114167 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.114070 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:06.471755 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.471727 4312 generic.go:356] "Generic (PLEG): container finished" podID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerID="f9eb66b79e3b58587644178d2ac38c5a9e6843cf2d715e220049ee8ede4173b9" exitCode=60 Nov 02 12:15:06.472202 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.471805 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" event={"ID":"4d2029df-8360-4cb6-925b-5622b6a3178d","Type":"ContainerDied","Data":"f9eb66b79e3b58587644178d2ac38c5a9e6843cf2d715e220049ee8ede4173b9"} Nov 02 12:15:06.472202 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.471858 4312 scope.go:122] "RemoveContainer" containerID="37d8ed181939a9428f7b1413aaa20b73475ed4eacca4787e079f0ed7d117e36a" Nov 02 12:15:06.472202 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.472076 4312 scope.go:122] "RemoveContainer" containerID="f9eb66b79e3b58587644178d2ac38c5a9e6843cf2d715e220049ee8ede4173b9" Nov 02 12:15:06.472378 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:15:06.472353 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq_odh-ai-gateway-infra(4d2029df-8360-4cb6-925b-5622b6a3178d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" Nov 02 12:15:06.816260 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.816180 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm"] Nov 02 12:15:06.819220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.819201 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:06.871241 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.871221 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 12:15:06.878207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.878185 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm"] Nov 02 12:15:06.937716 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.937687 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42cb2ce7-88nlm\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:06.937871 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.937855 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xlxk\" (UniqueName: \"kubernetes.io/projected/6b649583-6f05-4a86-9e8b-92c650a964c5-kube-api-access-5xlxk\") pod \"maas-api-revoke-sub-42cb2ce7-88nlm\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:06.937965 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:06.937950 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42cb2ce7-88nlm\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:07.038893 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:07.038870 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42cb2ce7-88nlm\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:07.039021 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:07.038933 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xlxk\" (UniqueName: \"kubernetes.io/projected/6b649583-6f05-4a86-9e8b-92c650a964c5-kube-api-access-5xlxk\") pod \"maas-api-revoke-sub-42cb2ce7-88nlm\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:07.039021 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:07.038970 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42cb2ce7-88nlm\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:07.039570 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:07.039549 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42cb2ce7-88nlm\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:07.041160 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:07.041140 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42cb2ce7-88nlm\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:07.100098 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:07.100039 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xlxk\" (UniqueName: \"kubernetes.io/projected/6b649583-6f05-4a86-9e8b-92c650a964c5-kube-api-access-5xlxk\") pod \"maas-api-revoke-sub-42cb2ce7-88nlm\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:07.130810 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:07.130781 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:07.490519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:07.489596 4312 scope.go:122] "RemoveContainer" containerID="f9eb66b79e3b58587644178d2ac38c5a9e6843cf2d715e220049ee8ede4173b9" Nov 02 12:15:07.490519 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:15:07.489872 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq_odh-ai-gateway-infra(4d2029df-8360-4cb6-925b-5622b6a3178d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" Nov 02 12:15:07.512279 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:07.512251 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm"] Nov 02 12:15:07.513852 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:15:07.513822 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b649583_6f05_4a86_9e8b_92c650a964c5.slice/crio-0f06cc5e117a2c760e703e1f5a7b9373dea1bf2d62bcd45e10f7c3695c85b27f WatchSource:0}: Error finding container 0f06cc5e117a2c760e703e1f5a7b9373dea1bf2d62bcd45e10f7c3695c85b27f: Status 404 returned error can't find the container with id 0f06cc5e117a2c760e703e1f5a7b9373dea1bf2d62bcd45e10f7c3695c85b27f Nov 02 12:15:08.495587 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:08.495560 4312 generic.go:356] "Generic (PLEG): container finished" podID="6b649583-6f05-4a86-9e8b-92c650a964c5" containerID="b34b93387d3f7ae562d5720fe3a259113ae648a0cfae3d45f395a39c6e57c2ea" exitCode=0 Nov 02 12:15:08.495937 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:08.495643 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" event={"ID":"6b649583-6f05-4a86-9e8b-92c650a964c5","Type":"ContainerDied","Data":"b34b93387d3f7ae562d5720fe3a259113ae648a0cfae3d45f395a39c6e57c2ea"} Nov 02 12:15:08.495937 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:08.495692 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" event={"ID":"6b649583-6f05-4a86-9e8b-92c650a964c5","Type":"ContainerStarted","Data":"0f06cc5e117a2c760e703e1f5a7b9373dea1bf2d62bcd45e10f7c3695c85b27f"} Nov 02 12:15:09.646913 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:09.646889 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:09.768847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:09.768787 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-openshift-ca\") pod \"6b649583-6f05-4a86-9e8b-92c650a964c5\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " Nov 02 12:15:09.768847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:09.768836 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b649583-6f05-4a86-9e8b-92c650a964c5-kube-api-access-5xlxk\" (UniqueName: \"kubernetes.io/projected/6b649583-6f05-4a86-9e8b-92c650a964c5-kube-api-access-5xlxk\") pod \"6b649583-6f05-4a86-9e8b-92c650a964c5\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " Nov 02 12:15:09.769002 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:09.768872 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-xks-ca\") pod \"6b649583-6f05-4a86-9e8b-92c650a964c5\" (UID: \"6b649583-6f05-4a86-9e8b-92c650a964c5\") " Nov 02 12:15:09.769243 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:09.769216 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-openshift-ca" pod "6b649583-6f05-4a86-9e8b-92c650a964c5" (UID: "6b649583-6f05-4a86-9e8b-92c650a964c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:15:09.771098 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:09.771072 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-xks-ca" pod "6b649583-6f05-4a86-9e8b-92c650a964c5" (UID: "6b649583-6f05-4a86-9e8b-92c650a964c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:15:09.771223 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:09.771153 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b649583-6f05-4a86-9e8b-92c650a964c5-kube-api-access-5xlxk" pod "6b649583-6f05-4a86-9e8b-92c650a964c5" (UID: "6b649583-6f05-4a86-9e8b-92c650a964c5"). InnerVolumeSpecName "kube-api-access-5xlxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:15:09.782701 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:09.782675 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:09.869978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:09.869955 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-xks-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:09.869978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:09.869977 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b649583-6f05-4a86-9e8b-92c650a964c5-maas-api-openshift-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:09.870107 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:09.869989 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xlxk\" (UniqueName: \"kubernetes.io/projected/6b649583-6f05-4a86-9e8b-92c650a964c5-kube-api-access-5xlxk\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:10.506631 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:10.506602 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" Nov 02 12:15:10.506631 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:10.506619 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm" event={"ID":"6b649583-6f05-4a86-9e8b-92c650a964c5","Type":"ContainerDied","Data":"0f06cc5e117a2c760e703e1f5a7b9373dea1bf2d62bcd45e10f7c3695c85b27f"} Nov 02 12:15:10.506902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:10.506661 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f06cc5e117a2c760e703e1f5a7b9373dea1bf2d62bcd45e10f7c3695c85b27f" Nov 02 12:15:11.090439 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:11.090408 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:14.773237 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.773194 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg"] Nov 02 12:15:14.773951 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.773927 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6b649583-6f05-4a86-9e8b-92c650a964c5" containerName="revoke-keys" Nov 02 12:15:14.774032 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.773953 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b649583-6f05-4a86-9e8b-92c650a964c5" containerName="revoke-keys" Nov 02 12:15:14.774086 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.774062 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b649583-6f05-4a86-9e8b-92c650a964c5" containerName="revoke-keys" Nov 02 12:15:14.778214 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.778189 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:14.903314 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.903256 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg"] Nov 02 12:15:14.924793 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.924753 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:14.924971 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.924823 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:14.924971 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.924896 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:14.924971 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.924935 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmljj\" (UniqueName: \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-kube-api-access-zmljj\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:14.925161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.924977 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:14.925161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.925026 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:14.925161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.925096 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:14.925325 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.925158 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:14.925325 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:14.925201 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.026237 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.026127 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.026574 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.026552 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.026790 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.026756 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.026946 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.026929 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.027107 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.027091 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.027248 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.027230 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.027450 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.027433 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.027657 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.027623 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmljj\" (UniqueName: \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-kube-api-access-zmljj\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.027817 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.027800 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.028018 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.026881 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.028145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.027549 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istiod-ca-cert\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.028264 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.026645 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-credential-socket\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.028748 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.028726 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-data\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.028883 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.028780 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-certs\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.031697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.031652 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-podinfo\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.032380 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.032336 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-envoy\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.089286 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.089234 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmljj\" (UniqueName: \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-kube-api-access-zmljj\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.099069 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.099017 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-token\") pod \"e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:15.401980 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:15.401665 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:16.520573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:16.502989 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:16.576722 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:16.576681 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:15:16.583526 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:16.579807 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg"] Nov 02 12:15:16.625515 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:16.624571 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:15:16.659967 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:16.659597 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" podUID="67d52846-b9e0-4b0c-b03e-f2e90b668969" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:15:16.750466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:16.750416 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:15:16.750706 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:16.750684 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:15:16.750913 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:16.750877 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:15:16.953874 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:16.953579 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:15:17.268528 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:17.255627 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb"] Nov 02 12:15:17.268528 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:17.255883 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" containerID="cri-o://13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d" gracePeriod=30 Nov 02 12:15:17.636383 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:17.636278 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" event={"ID":"6f6435ad-c2f0-4586-a026-c4f57abcbb05","Type":"ContainerStarted","Data":"a3ac1e6bd18de0b17e7b47bb276f5b5a3ffff6e80d2a052e6cd699bb59bde5eb"} Nov 02 12:15:17.636383 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:17.636343 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" event={"ID":"6f6435ad-c2f0-4586-a026-c4f57abcbb05","Type":"ContainerStarted","Data":"f9148dca53815911c37876fa467db994700671d510059164703b0cbb0eaa29f0"} Nov 02 12:15:17.759986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:17.759937 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" podStartSLOduration=3.7599236019999998 podStartE2EDuration="3.759923602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:15:17.756295554 +0000 UTC m=+1134.786531807" watchObservedRunningTime="2026-11-02 12:15:17.759923602 +0000 UTC m=+1134.790159852" Nov 02 12:15:18.402764 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:18.402731 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:18.404153 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:18.404118 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 12:15:18.404305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:18.404170 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" podUID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 12:15:19.069521 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:19.069104 4312 scope.go:122] "RemoveContainer" containerID="f9eb66b79e3b58587644178d2ac38c5a9e6843cf2d715e220049ee8ede4173b9" Nov 02 12:15:19.402947 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:19.402921 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 12:15:19.403062 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:19.402963 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" podUID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 12:15:19.676727 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:19.676695 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" event={"ID":"4d2029df-8360-4cb6-925b-5622b6a3178d","Type":"ContainerStarted","Data":"f381e44d7f111fdddb411ae6fb5ad2d1f04934b73934b9cebac2bd991fd05634"} Nov 02 12:15:19.793830 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:19.793768 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" podStartSLOduration=16.399874412 podStartE2EDuration="19.793749155s" totalImagesPullingTime="3.393874743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:15:19.792279274 +0000 UTC m=+1136.822515525" watchObservedRunningTime="2026-11-02 12:15:19.793749155 +0000 UTC m=+1136.823985406" Nov 02 12:15:20.403240 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:20.403166 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 12:15:20.403240 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:20.403218 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" podUID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 12:15:20.683420 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:20.683391 4312 generic.go:356] "Generic (PLEG): container finished" podID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerID="f381e44d7f111fdddb411ae6fb5ad2d1f04934b73934b9cebac2bd991fd05634" exitCode=60 Nov 02 12:15:20.683604 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:20.683450 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" event={"ID":"4d2029df-8360-4cb6-925b-5622b6a3178d","Type":"ContainerDied","Data":"f381e44d7f111fdddb411ae6fb5ad2d1f04934b73934b9cebac2bd991fd05634"} Nov 02 12:15:20.683604 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:20.683536 4312 scope.go:122] "RemoveContainer" containerID="f9eb66b79e3b58587644178d2ac38c5a9e6843cf2d715e220049ee8ede4173b9" Nov 02 12:15:20.683937 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:20.683914 4312 scope.go:122] "RemoveContainer" containerID="f381e44d7f111fdddb411ae6fb5ad2d1f04934b73934b9cebac2bd991fd05634" Nov 02 12:15:20.684228 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:15:20.684203 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq_odh-ai-gateway-infra(4d2029df-8360-4cb6-925b-5622b6a3178d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" Nov 02 12:15:20.804343 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:20.804314 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq"] Nov 02 12:15:21.402773 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:21.402740 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 12:15:21.402944 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:21.402799 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" podUID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 12:15:21.408575 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:21.408550 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:21.846048 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:21.846025 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" Nov 02 12:15:21.996984 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:21.996909 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d2029df-8360-4cb6-925b-5622b6a3178d-kube-api-access-xqw5p\" (UniqueName: \"kubernetes.io/projected/4d2029df-8360-4cb6-925b-5622b6a3178d-kube-api-access-xqw5p\") pod \"4d2029df-8360-4cb6-925b-5622b6a3178d\" (UID: \"4d2029df-8360-4cb6-925b-5622b6a3178d\") " Nov 02 12:15:21.997156 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:21.996996 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d2029df-8360-4cb6-925b-5622b6a3178d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2029df-8360-4cb6-925b-5622b6a3178d-maas-api-service-ca\") pod \"4d2029df-8360-4cb6-925b-5622b6a3178d\" (UID: \"4d2029df-8360-4cb6-925b-5622b6a3178d\") " Nov 02 12:15:21.997403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:21.997375 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2029df-8360-4cb6-925b-5622b6a3178d-maas-api-service-ca" pod "4d2029df-8360-4cb6-925b-5622b6a3178d" (UID: "4d2029df-8360-4cb6-925b-5622b6a3178d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:15:21.997536 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:21.997518 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2029df-8360-4cb6-925b-5622b6a3178d-maas-api-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:21.999353 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:21.999329 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2029df-8360-4cb6-925b-5622b6a3178d-kube-api-access-xqw5p" pod "4d2029df-8360-4cb6-925b-5622b6a3178d" (UID: "4d2029df-8360-4cb6-925b-5622b6a3178d"). InnerVolumeSpecName "kube-api-access-xqw5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:15:22.098273 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.098247 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqw5p\" (UniqueName: \"kubernetes.io/projected/4d2029df-8360-4cb6-925b-5622b6a3178d-kube-api-access-xqw5p\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:22.403331 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.403304 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 12:15:22.403449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.403362 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" podUID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 12:15:22.639204 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.639176 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:15:22.696930 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.696904 4312 generic.go:356] "Generic (PLEG): container finished" podID="684df0a9-463d-4c13-bbcd-33150e90687d" containerID="13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d" exitCode=0 Nov 02 12:15:22.697085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.696949 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" Nov 02 12:15:22.697085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.696986 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" event={"ID":"684df0a9-463d-4c13-bbcd-33150e90687d","Type":"ContainerDied","Data":"13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d"} Nov 02 12:15:22.697085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.697042 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb" event={"ID":"684df0a9-463d-4c13-bbcd-33150e90687d","Type":"ContainerDied","Data":"8c92de8ed6b4a689ac85d96654e62814580f7a5a4eb7afba4a0511bb0d33628a"} Nov 02 12:15:22.697085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.697063 4312 scope.go:122] "RemoveContainer" containerID="13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d" Nov 02 12:15:22.698466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.698449 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" event={"ID":"4d2029df-8360-4cb6-925b-5622b6a3178d","Type":"ContainerDied","Data":"29836896509f5e367b349784c0387769e4a3401a25b2a62e97edf0c3afc95d77"} Nov 02 12:15:22.698466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.698465 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq" Nov 02 12:15:22.705858 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.705841 4312 scope.go:122] "RemoveContainer" containerID="13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d" Nov 02 12:15:22.706197 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:15:22.706123 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d\": container with ID starting with 13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d not found: ID does not exist" containerID="13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d" Nov 02 12:15:22.706197 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.706162 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d"} err="failed to get container status \"13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d\": rpc error: code = NotFound desc = could not find container \"13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d\": container with ID starting with 13745d040d4af25ac57399a1733bb757cb3779aad21aeec898c715eb38cf2a8d not found: ID does not exist" Nov 02 12:15:22.706197 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.706185 4312 scope.go:122] "RemoveContainer" containerID="f381e44d7f111fdddb411ae6fb5ad2d1f04934b73934b9cebac2bd991fd05634" Nov 02 12:15:22.806674 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.806642 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/684df0a9-463d-4c13-bbcd-33150e90687d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/684df0a9-463d-4c13-bbcd-33150e90687d-istiod-ca-cert\") pod \"684df0a9-463d-4c13-bbcd-33150e90687d\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " Nov 02 12:15:22.806846 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.806702 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-certs\") pod \"684df0a9-463d-4c13-bbcd-33150e90687d\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " Nov 02 12:15:22.806846 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.806743 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-credential-socket\") pod \"684df0a9-463d-4c13-bbcd-33150e90687d\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " Nov 02 12:15:22.806846 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.806776 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-data\") pod \"684df0a9-463d-4c13-bbcd-33150e90687d\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " Nov 02 12:15:22.807150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.806855 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-istio-token\" (UniqueName: \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-istio-token\") pod \"684df0a9-463d-4c13-bbcd-33150e90687d\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " Nov 02 12:15:22.807150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.806878 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-kube-api-access-dfvj7\" (UniqueName: \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-kube-api-access-dfvj7\") pod \"684df0a9-463d-4c13-bbcd-33150e90687d\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " Nov 02 12:15:22.807150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.806901 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-envoy\") pod \"684df0a9-463d-4c13-bbcd-33150e90687d\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " Nov 02 12:15:22.807150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.806937 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/684df0a9-463d-4c13-bbcd-33150e90687d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/684df0a9-463d-4c13-bbcd-33150e90687d-istio-podinfo\") pod \"684df0a9-463d-4c13-bbcd-33150e90687d\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " Nov 02 12:15:22.807150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.806977 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-socket\") pod \"684df0a9-463d-4c13-bbcd-33150e90687d\" (UID: \"684df0a9-463d-4c13-bbcd-33150e90687d\") " Nov 02 12:15:22.807545 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.807521 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-data" pod "684df0a9-463d-4c13-bbcd-33150e90687d" (UID: "684df0a9-463d-4c13-bbcd-33150e90687d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:15:22.807731 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.807703 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-certs" pod "684df0a9-463d-4c13-bbcd-33150e90687d" (UID: "684df0a9-463d-4c13-bbcd-33150e90687d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:15:22.807803 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.807714 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-credential-socket" pod "684df0a9-463d-4c13-bbcd-33150e90687d" (UID: "684df0a9-463d-4c13-bbcd-33150e90687d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:15:22.807803 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.807765 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684df0a9-463d-4c13-bbcd-33150e90687d-istiod-ca-cert" pod "684df0a9-463d-4c13-bbcd-33150e90687d" (UID: "684df0a9-463d-4c13-bbcd-33150e90687d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:15:22.807988 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.807967 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-socket" pod "684df0a9-463d-4c13-bbcd-33150e90687d" (UID: "684df0a9-463d-4c13-bbcd-33150e90687d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:15:22.809886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.809834 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-istio-token" pod "684df0a9-463d-4c13-bbcd-33150e90687d" (UID: "684df0a9-463d-4c13-bbcd-33150e90687d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:15:22.810186 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.810159 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-envoy" pod "684df0a9-463d-4c13-bbcd-33150e90687d" (UID: "684df0a9-463d-4c13-bbcd-33150e90687d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:15:22.810258 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.810218 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/684df0a9-463d-4c13-bbcd-33150e90687d-istio-podinfo" pod "684df0a9-463d-4c13-bbcd-33150e90687d" (UID: "684df0a9-463d-4c13-bbcd-33150e90687d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:15:22.810324 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.810311 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-kube-api-access-dfvj7" pod "684df0a9-463d-4c13-bbcd-33150e90687d" (UID: "684df0a9-463d-4c13-bbcd-33150e90687d"). InnerVolumeSpecName "kube-api-access-dfvj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:15:22.908703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.908667 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:22.908703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.908705 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:22.908905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.908719 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:22.908905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.908734 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:22.908905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.908749 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfvj7\" (UniqueName: \"kubernetes.io/projected/684df0a9-463d-4c13-bbcd-33150e90687d-kube-api-access-dfvj7\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:22.908905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.908762 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:22.908905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.908775 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/684df0a9-463d-4c13-bbcd-33150e90687d-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:22.908905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.908790 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/684df0a9-463d-4c13-bbcd-33150e90687d-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:22.908905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.908804 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/684df0a9-463d-4c13-bbcd-33150e90687d-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:22.966674 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.966548 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq"] Nov 02 12:15:22.974868 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:22.974842 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7bd72e-29893695-rh7jq"] Nov 02 12:15:23.023315 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:15:23.023288 4312 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684df0a9_463d_4c13_bbcd_33150e90687d.slice/crio-8c92de8ed6b4a689ac85d96654e62814580f7a5a4eb7afba4a0511bb0d33628a\": RecentStats: unable to find data in memory cache]" Nov 02 12:15:23.079397 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:23.079370 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" path="/var/lib/kubelet/pods/4d2029df-8360-4cb6-925b-5622b6a3178d/volumes" Nov 02 12:15:23.133645 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:23.133620 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb"] Nov 02 12:15:23.138018 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:23.137994 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cjqgb"] Nov 02 12:15:23.402615 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:23.402533 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 02 12:15:23.402777 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:23.402605 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" podUID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 02 12:15:24.508827 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:24.508789 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:24.509599 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:24.509570 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:24.510709 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:24.510689 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:15:25.076071 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:25.076036 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" path="/var/lib/kubelet/pods/684df0a9-463d-4c13-bbcd-33150e90687d/volumes" Nov 02 12:15:26.232635 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:26.232606 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:26.482538 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:26.482485 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:29.594147 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:15:29.594080 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:15:30.514644 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:30.514600 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" podUID="f01b2ae0-f424-46de-976f-fd2188cbfb6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:15:30.612628 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:30.612583 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:15:30.664630 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:30.664578 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" podUID="67d52846-b9e0-4b0c-b03e-f2e90b668969" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:15:31.982376 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:31.982343 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:36.895686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:36.895639 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:37.768885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.768853 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7227-2f49b6"] Nov 02 12:15:37.769262 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769249 4312 cpu_manager.go:414] "containerMap: removing container" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" Nov 02 12:15:37.769319 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769263 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" Nov 02 12:15:37.769319 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769279 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerName="cleanup" Nov 02 12:15:37.769319 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769285 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerName="cleanup" Nov 02 12:15:37.769319 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769303 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerName="cleanup" Nov 02 12:15:37.769319 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769309 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerName="cleanup" Nov 02 12:15:37.769319 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769317 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerName="cleanup" Nov 02 12:15:37.769319 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769321 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerName="cleanup" Nov 02 12:15:37.769588 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769380 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerName="cleanup" Nov 02 12:15:37.769588 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769388 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerName="cleanup" Nov 02 12:15:37.769588 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769396 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="684df0a9-463d-4c13-bbcd-33150e90687d" containerName="istio-proxy" Nov 02 12:15:37.769588 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.769403 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d2029df-8360-4cb6-925b-5622b6a3178d" containerName="cleanup" Nov 02 12:15:37.773915 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.773895 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-2f49b6" Nov 02 12:15:37.826990 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.826965 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-2f49b6"] Nov 02 12:15:37.873726 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.873702 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5lqpk\"" Nov 02 12:15:37.906405 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:37.906378 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twcjt\" (UniqueName: \"kubernetes.io/projected/0ada8a7b-e117-4583-ac7e-b0cc14c959c0-kube-api-access-twcjt\") pod \"test-curl-7227-2f49b6\" (UID: \"0ada8a7b-e117-4583-ac7e-b0cc14c959c0\") " pod="opendatahub/test-curl-7227-2f49b6" Nov 02 12:15:38.007246 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:38.007216 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twcjt\" (UniqueName: \"kubernetes.io/projected/0ada8a7b-e117-4583-ac7e-b0cc14c959c0-kube-api-access-twcjt\") pod \"test-curl-7227-2f49b6\" (UID: \"0ada8a7b-e117-4583-ac7e-b0cc14c959c0\") " pod="opendatahub/test-curl-7227-2f49b6" Nov 02 12:15:38.068107 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:38.068041 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twcjt\" (UniqueName: \"kubernetes.io/projected/0ada8a7b-e117-4583-ac7e-b0cc14c959c0-kube-api-access-twcjt\") pod \"test-curl-7227-2f49b6\" (UID: \"0ada8a7b-e117-4583-ac7e-b0cc14c959c0\") " pod="opendatahub/test-curl-7227-2f49b6" Nov 02 12:15:38.085109 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:38.085087 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-2f49b6" Nov 02 12:15:38.457218 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:38.457194 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-2f49b6"] Nov 02 12:15:38.459038 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:15:38.459013 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ada8a7b_e117_4583_ac7e_b0cc14c959c0.slice/crio-95bc59668f4d21b52fe708549461b4a10deb63ef010e32b2d5393009851e0130 WatchSource:0}: Error finding container 95bc59668f4d21b52fe708549461b4a10deb63ef010e32b2d5393009851e0130: Status 404 returned error can't find the container with id 95bc59668f4d21b52fe708549461b4a10deb63ef010e32b2d5393009851e0130 Nov 02 12:15:38.811781 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:38.811710 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-2f49b6" event={"ID":"0ada8a7b-e117-4583-ac7e-b0cc14c959c0","Type":"ContainerStarted","Data":"95bc59668f4d21b52fe708549461b4a10deb63ef010e32b2d5393009851e0130"} Nov 02 12:15:41.814660 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:41.814625 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:41.831151 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:41.831125 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-2f49b6" event={"ID":"0ada8a7b-e117-4583-ac7e-b0cc14c959c0","Type":"ContainerStarted","Data":"2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee"} Nov 02 12:15:41.947103 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:41.947057 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7227-2f49b6" podStartSLOduration=2.584325087 podStartE2EDuration="4.947042568s" totalImagesPullingTime="2.362717481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:15:41.945759852 +0000 UTC m=+1158.975996101" watchObservedRunningTime="2026-11-02 12:15:41.947042568 +0000 UTC m=+1158.977279202" Nov 02 12:15:43.439710 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:43.439601 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7227-2f49b6"] Nov 02 12:15:43.444835 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:43.442700 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7227-2f49b6"] Nov 02 12:15:43.841353 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:43.841267 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7227-2f49b6" podUID="0ada8a7b-e117-4583-ac7e-b0cc14c959c0" containerName="test-curl-7227-2f49b6" containerID="cri-o://2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee" gracePeriod=2 Nov 02 12:15:43.892243 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:15:43.892206 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-2f49b6\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="0ada8a7b-e117-4583-ac7e-b0cc14c959c0" pod="opendatahub/test-curl-7227-2f49b6" Nov 02 12:15:46.724859 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:46.724816 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-2f49b6" Nov 02 12:15:46.929529 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:46.928478 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ada8a7b-e117-4583-ac7e-b0cc14c959c0-kube-api-access-twcjt\" (UniqueName: \"kubernetes.io/projected/0ada8a7b-e117-4583-ac7e-b0cc14c959c0-kube-api-access-twcjt\") pod \"0ada8a7b-e117-4583-ac7e-b0cc14c959c0\" (UID: \"0ada8a7b-e117-4583-ac7e-b0cc14c959c0\") " Nov 02 12:15:46.969532 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:46.965702 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ada8a7b-e117-4583-ac7e-b0cc14c959c0-kube-api-access-twcjt" pod "0ada8a7b-e117-4583-ac7e-b0cc14c959c0" (UID: "0ada8a7b-e117-4583-ac7e-b0cc14c959c0"). InnerVolumeSpecName "kube-api-access-twcjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:15:47.036199 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:47.036169 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twcjt\" (UniqueName: \"kubernetes.io/projected/0ada8a7b-e117-4583-ac7e-b0cc14c959c0-kube-api-access-twcjt\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:15:47.066435 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:47.061647 4312 generic.go:356] "Generic (PLEG): container finished" podID="0ada8a7b-e117-4583-ac7e-b0cc14c959c0" containerID="2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee" exitCode=137 Nov 02 12:15:47.066435 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:47.061710 4312 scope.go:122] "RemoveContainer" containerID="2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee" Nov 02 12:15:47.066435 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:47.061860 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-2f49b6" Nov 02 12:15:47.153872 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:47.153846 4312 scope.go:122] "RemoveContainer" containerID="2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee" Nov 02 12:15:47.162427 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:15:47.157960 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee\": container with ID starting with 2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee not found: ID does not exist" containerID="2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee" Nov 02 12:15:47.162427 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:47.160584 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee"} err="failed to get container status \"2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee\": rpc error: code = NotFound desc = could not find container \"2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee\": container with ID starting with 2a9bf73c0077d388dc47efea461fc56ab1f6f72a9f3336b6063b5db078250bee not found: ID does not exist" Nov 02 12:15:47.168260 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:15:47.166800 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-2f49b6\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="0ada8a7b-e117-4583-ac7e-b0cc14c959c0" pod="opendatahub/test-curl-7227-2f49b6" Nov 02 12:15:47.179746 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:47.179680 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ada8a7b-e117-4583-ac7e-b0cc14c959c0" path="/var/lib/kubelet/pods/0ada8a7b-e117-4583-ac7e-b0cc14c959c0/volumes" Nov 02 12:15:47.478964 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:47.478866 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:52.414136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:52.414104 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:57.411846 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:15:57.411812 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:15:59.555875 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:15:59.555835 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:16:02.825877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:02.825808 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:07.801772 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:07.801740 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:10.125969 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:10.125931 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:10.376675 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:10.376583 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:12.702836 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:12.702804 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:13.353039 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:13.352994 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:14.618150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:14.618116 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:15.540438 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:15.540402 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:15.997531 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:15.997402 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:17.556419 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:17.554689 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" podUID="f01b2ae0-f424-46de-976f-fd2188cbfb6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 12:16:17.571648 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:17.571578 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:16:17.604650 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:17.604582 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:16:17.658347 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:17.657604 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" podUID="67d52846-b9e0-4b0c-b03e-f2e90b668969" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Nov 02 12:16:17.933518 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:17.926563 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Nov 02 12:16:18.564079 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:18.564037 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:20.979389 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:20.979359 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:21.233599 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:21.233484 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:23.886610 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:23.886576 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:24.274693 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:24.274662 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:24.527616 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:24.527434 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:28.808377 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:28.808329 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:29.372403 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:29.372370 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:16:30.760204 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:30.760164 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9"] Nov 02 12:16:30.866072 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:30.866037 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7227-34e6b1"] Nov 02 12:16:30.866691 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:30.866670 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0ada8a7b-e117-4583-ac7e-b0cc14c959c0" containerName="test-curl-7227-2f49b6" Nov 02 12:16:30.866825 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:30.866693 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ada8a7b-e117-4583-ac7e-b0cc14c959c0" containerName="test-curl-7227-2f49b6" Nov 02 12:16:30.866891 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:30.866842 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ada8a7b-e117-4583-ac7e-b0cc14c959c0" containerName="test-curl-7227-2f49b6" Nov 02 12:16:30.874373 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:30.874348 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-34e6b1" Nov 02 12:16:30.895222 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:30.895191 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:30.925839 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:30.925810 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-34e6b1"] Nov 02 12:16:30.927143 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:30.927119 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5lqpk\"" Nov 02 12:16:30.959093 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:30.959054 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz5x2\" (UniqueName: \"kubernetes.io/projected/8365c444-2e64-461a-a4d0-e754a52fd615-kube-api-access-bz5x2\") pod \"test-curl-7227-34e6b1\" (UID: \"8365c444-2e64-461a-a4d0-e754a52fd615\") " pod="opendatahub/test-curl-7227-34e6b1" Nov 02 12:16:31.060728 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:31.060645 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz5x2\" (UniqueName: \"kubernetes.io/projected/8365c444-2e64-461a-a4d0-e754a52fd615-kube-api-access-bz5x2\") pod \"test-curl-7227-34e6b1\" (UID: \"8365c444-2e64-461a-a4d0-e754a52fd615\") " pod="opendatahub/test-curl-7227-34e6b1" Nov 02 12:16:31.121064 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:31.121029 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz5x2\" (UniqueName: \"kubernetes.io/projected/8365c444-2e64-461a-a4d0-e754a52fd615-kube-api-access-bz5x2\") pod \"test-curl-7227-34e6b1\" (UID: \"8365c444-2e64-461a-a4d0-e754a52fd615\") " pod="opendatahub/test-curl-7227-34e6b1" Nov 02 12:16:31.192091 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:31.192066 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-34e6b1" Nov 02 12:16:31.566891 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:31.566870 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-34e6b1"] Nov 02 12:16:31.568556 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:16:31.568527 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8365c444_2e64_461a_a4d0_e754a52fd615.slice/crio-e77aa6a65d7897a393f1e4ccef9ed7c114582f16a74a153ea25c6fb921aea9e7 WatchSource:0}: Error finding container e77aa6a65d7897a393f1e4ccef9ed7c114582f16a74a153ea25c6fb921aea9e7: Status 404 returned error can't find the container with id e77aa6a65d7897a393f1e4ccef9ed7c114582f16a74a153ea25c6fb921aea9e7 Nov 02 12:16:32.387810 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:32.387767 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-34e6b1" event={"ID":"8365c444-2e64-461a-a4d0-e754a52fd615","Type":"ContainerStarted","Data":"ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529"} Nov 02 12:16:32.387810 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:32.387813 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-34e6b1" event={"ID":"8365c444-2e64-461a-a4d0-e754a52fd615","Type":"ContainerStarted","Data":"e77aa6a65d7897a393f1e4ccef9ed7c114582f16a74a153ea25c6fb921aea9e7"} Nov 02 12:16:32.499931 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:32.499879 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7227-34e6b1" podStartSLOduration=2.499867692 podStartE2EDuration="2.499867692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:16:32.49957491 +0000 UTC m=+1209.529811160" watchObservedRunningTime="2026-11-02 12:16:32.499867692 +0000 UTC m=+1209.530103941" Nov 02 12:16:33.694768 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:33.694737 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:33.881488 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:33.881451 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7227-34e6b1"] Nov 02 12:16:33.885888 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:33.885864 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7227-34e6b1"] Nov 02 12:16:34.034687 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.034614 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7227-97f775"] Nov 02 12:16:34.035063 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.035050 4312 cpu_manager.go:414] "containerMap: removing container" podUID="8365c444-2e64-461a-a4d0-e754a52fd615" containerName="test-curl-7227-34e6b1" Nov 02 12:16:34.035106 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.035064 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8365c444-2e64-461a-a4d0-e754a52fd615" containerName="test-curl-7227-34e6b1" Nov 02 12:16:34.035147 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.035136 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="8365c444-2e64-461a-a4d0-e754a52fd615" containerName="test-curl-7227-34e6b1" Nov 02 12:16:34.038314 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.038296 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-97f775" Nov 02 12:16:34.089928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.089902 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xggdj\" (UniqueName: \"kubernetes.io/projected/20b1f1b9-6a37-4e7e-b4ba-9c472d322858-kube-api-access-xggdj\") pod \"test-curl-7227-97f775\" (UID: \"20b1f1b9-6a37-4e7e-b4ba-9c472d322858\") " pod="opendatahub/test-curl-7227-97f775" Nov 02 12:16:34.095169 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.095143 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-97f775"] Nov 02 12:16:34.191441 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.191415 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xggdj\" (UniqueName: \"kubernetes.io/projected/20b1f1b9-6a37-4e7e-b4ba-9c472d322858-kube-api-access-xggdj\") pod \"test-curl-7227-97f775\" (UID: \"20b1f1b9-6a37-4e7e-b4ba-9c472d322858\") " pod="opendatahub/test-curl-7227-97f775" Nov 02 12:16:34.252456 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.252432 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xggdj\" (UniqueName: \"kubernetes.io/projected/20b1f1b9-6a37-4e7e-b4ba-9c472d322858-kube-api-access-xggdj\") pod \"test-curl-7227-97f775\" (UID: \"20b1f1b9-6a37-4e7e-b4ba-9c472d322858\") " pod="opendatahub/test-curl-7227-97f775" Nov 02 12:16:34.350477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.350422 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-97f775" Nov 02 12:16:34.402992 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.396941 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7227-34e6b1" podUID="8365c444-2e64-461a-a4d0-e754a52fd615" containerName="test-curl-7227-34e6b1" containerID="cri-o://ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529" gracePeriod=2 Nov 02 12:16:34.447938 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:34.447900 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-34e6b1\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="8365c444-2e64-461a-a4d0-e754a52fd615" pod="opendatahub/test-curl-7227-34e6b1" Nov 02 12:16:34.721472 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:34.721440 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-97f775"] Nov 02 12:16:34.723908 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:16:34.723885 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20b1f1b9_6a37_4e7e_b4ba_9c472d322858.slice/crio-9e9dd8e0caa448dd86f27be9962fc7e08915d5c640309ad675d389706f84dacb WatchSource:0}: Error finding container 9e9dd8e0caa448dd86f27be9962fc7e08915d5c640309ad675d389706f84dacb: Status 404 returned error can't find the container with id 9e9dd8e0caa448dd86f27be9962fc7e08915d5c640309ad675d389706f84dacb Nov 02 12:16:35.404013 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:35.403978 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-97f775" event={"ID":"20b1f1b9-6a37-4e7e-b4ba-9c472d322858","Type":"ContainerStarted","Data":"90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031"} Nov 02 12:16:35.404013 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:35.404022 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-97f775" event={"ID":"20b1f1b9-6a37-4e7e-b4ba-9c472d322858","Type":"ContainerStarted","Data":"9e9dd8e0caa448dd86f27be9962fc7e08915d5c640309ad675d389706f84dacb"} Nov 02 12:16:35.520191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:35.520135 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7227-97f775" podStartSLOduration=2.520121236 podStartE2EDuration="2.520121236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:16:35.51793375 +0000 UTC m=+1212.548170001" watchObservedRunningTime="2026-11-02 12:16:35.520121236 +0000 UTC m=+1212.550357486" Nov 02 12:16:36.637579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:36.637552 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-34e6b1" Nov 02 12:16:36.715457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:36.715435 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8365c444-2e64-461a-a4d0-e754a52fd615-kube-api-access-bz5x2\" (UniqueName: \"kubernetes.io/projected/8365c444-2e64-461a-a4d0-e754a52fd615-kube-api-access-bz5x2\") pod \"8365c444-2e64-461a-a4d0-e754a52fd615\" (UID: \"8365c444-2e64-461a-a4d0-e754a52fd615\") " Nov 02 12:16:36.717307 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:36.717283 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8365c444-2e64-461a-a4d0-e754a52fd615-kube-api-access-bz5x2" pod "8365c444-2e64-461a-a4d0-e754a52fd615" (UID: "8365c444-2e64-461a-a4d0-e754a52fd615"). InnerVolumeSpecName "kube-api-access-bz5x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:16:36.816979 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:36.816954 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz5x2\" (UniqueName: \"kubernetes.io/projected/8365c444-2e64-461a-a4d0-e754a52fd615-kube-api-access-bz5x2\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:16:37.020915 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.020847 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7227-97f775"] Nov 02 12:16:37.023996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.023972 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7227-97f775"] Nov 02 12:16:37.075643 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.075614 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8365c444-2e64-461a-a4d0-e754a52fd615" path="/var/lib/kubelet/pods/8365c444-2e64-461a-a4d0-e754a52fd615/volumes" Nov 02 12:16:37.163065 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.163034 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7227-634ce0"] Nov 02 12:16:37.163527 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.163510 4312 cpu_manager.go:414] "containerMap: removing container" podUID="20b1f1b9-6a37-4e7e-b4ba-9c472d322858" containerName="test-curl-7227-97f775" Nov 02 12:16:37.163527 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.163526 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b1f1b9-6a37-4e7e-b4ba-9c472d322858" containerName="test-curl-7227-97f775" Nov 02 12:16:37.163650 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.163619 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b1f1b9-6a37-4e7e-b4ba-9c472d322858" containerName="test-curl-7227-97f775" Nov 02 12:16:37.166204 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.166186 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-634ce0" Nov 02 12:16:37.220617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.220598 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfg6h\" (UniqueName: \"kubernetes.io/projected/4d5e66b1-037f-45f2-ba71-e189766cae70-kube-api-access-dfg6h\") pod \"test-curl-7227-634ce0\" (UID: \"4d5e66b1-037f-45f2-ba71-e189766cae70\") " pod="opendatahub/test-curl-7227-634ce0" Nov 02 12:16:37.228893 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.228867 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-634ce0"] Nov 02 12:16:37.322241 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.322168 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfg6h\" (UniqueName: \"kubernetes.io/projected/4d5e66b1-037f-45f2-ba71-e189766cae70-kube-api-access-dfg6h\") pod \"test-curl-7227-634ce0\" (UID: \"4d5e66b1-037f-45f2-ba71-e189766cae70\") " pod="opendatahub/test-curl-7227-634ce0" Nov 02 12:16:37.382296 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.382270 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfg6h\" (UniqueName: \"kubernetes.io/projected/4d5e66b1-037f-45f2-ba71-e189766cae70-kube-api-access-dfg6h\") pod \"test-curl-7227-634ce0\" (UID: \"4d5e66b1-037f-45f2-ba71-e189766cae70\") " pod="opendatahub/test-curl-7227-634ce0" Nov 02 12:16:37.416564 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.416532 4312 generic.go:356] "Generic (PLEG): container finished" podID="8365c444-2e64-461a-a4d0-e754a52fd615" containerID="ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529" exitCode=137 Nov 02 12:16:37.416718 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.416619 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-34e6b1" Nov 02 12:16:37.416718 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.416637 4312 scope.go:122] "RemoveContainer" containerID="ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529" Nov 02 12:16:37.416840 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.416758 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7227-97f775" podUID="20b1f1b9-6a37-4e7e-b4ba-9c472d322858" containerName="test-curl-7227-97f775" containerID="cri-o://90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031" gracePeriod=2 Nov 02 12:16:37.427317 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.427291 4312 scope.go:122] "RemoveContainer" containerID="ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529" Nov 02 12:16:37.427730 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:37.427707 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529\": container with ID starting with ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529 not found: ID does not exist" containerID="ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529" Nov 02 12:16:37.427833 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.427741 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529"} err="failed to get container status \"ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529\": rpc error: code = NotFound desc = could not find container \"ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529\": container with ID starting with ffe240b0574561f4146921a66ced575c80ac23372e9b10279c059840a6c74529 not found: ID does not exist" Nov 02 12:16:37.478220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.478176 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-634ce0" Nov 02 12:16:37.964522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:37.963342 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-634ce0"] Nov 02 12:16:38.053521 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:16:38.038767 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d5e66b1_037f_45f2_ba71_e189766cae70.slice/crio-482fa4fd49b45598ab628929c8ec8d3053f6409ae1e5a34b55886143344385da WatchSource:0}: Error finding container 482fa4fd49b45598ab628929c8ec8d3053f6409ae1e5a34b55886143344385da: Status 404 returned error can't find the container with id 482fa4fd49b45598ab628929c8ec8d3053f6409ae1e5a34b55886143344385da Nov 02 12:16:38.445521 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:38.444652 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-634ce0" event={"ID":"4d5e66b1-037f-45f2-ba71-e189766cae70","Type":"ContainerStarted","Data":"482fa4fd49b45598ab628929c8ec8d3053f6409ae1e5a34b55886143344385da"} Nov 02 12:16:39.190526 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:39.181980 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:39.511116 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:39.508306 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" podUID="f01b2ae0-f424-46de-976f-fd2188cbfb6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:16:39.516519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:39.513213 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-634ce0" event={"ID":"4d5e66b1-037f-45f2-ba71-e189766cae70","Type":"ContainerStarted","Data":"031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21"} Nov 02 12:16:39.601296 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:39.600583 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:16:39.636516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:39.610568 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:16:39.706203 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:39.651277 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7227-634ce0" podStartSLOduration=2.6512566680000003 podStartE2EDuration="2.651256668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:16:39.647850247 +0000 UTC m=+1216.678086497" watchObservedRunningTime="2026-11-02 12:16:39.651256668 +0000 UTC m=+1216.681492921" Nov 02 12:16:39.764535 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:39.660893 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" podUID="67d52846-b9e0-4b0c-b03e-f2e90b668969" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:16:40.307687 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:40.307657 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-97f775" Nov 02 12:16:40.385716 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:40.385645 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b1f1b9-6a37-4e7e-b4ba-9c472d322858-kube-api-access-xggdj\" (UniqueName: \"kubernetes.io/projected/20b1f1b9-6a37-4e7e-b4ba-9c472d322858-kube-api-access-xggdj\") pod \"20b1f1b9-6a37-4e7e-b4ba-9c472d322858\" (UID: \"20b1f1b9-6a37-4e7e-b4ba-9c472d322858\") " Nov 02 12:16:40.387544 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:40.387516 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b1f1b9-6a37-4e7e-b4ba-9c472d322858-kube-api-access-xggdj" pod "20b1f1b9-6a37-4e7e-b4ba-9c472d322858" (UID: "20b1f1b9-6a37-4e7e-b4ba-9c472d322858"). InnerVolumeSpecName "kube-api-access-xggdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:16:40.487142 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:40.487118 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xggdj\" (UniqueName: \"kubernetes.io/projected/20b1f1b9-6a37-4e7e-b4ba-9c472d322858-kube-api-access-xggdj\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:16:40.520129 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:40.520100 4312 generic.go:356] "Generic (PLEG): container finished" podID="20b1f1b9-6a37-4e7e-b4ba-9c472d322858" containerID="90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031" exitCode=137 Nov 02 12:16:40.520220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:40.520144 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-97f775" Nov 02 12:16:40.520220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:40.520193 4312 scope.go:122] "RemoveContainer" containerID="90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031" Nov 02 12:16:40.529472 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:40.529447 4312 scope.go:122] "RemoveContainer" containerID="90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031" Nov 02 12:16:40.529730 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:40.529713 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031\": container with ID starting with 90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031 not found: ID does not exist" containerID="90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031" Nov 02 12:16:40.529781 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:40.529739 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031"} err="failed to get container status \"90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031\": rpc error: code = NotFound desc = could not find container \"90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031\": container with ID starting with 90df712448dbf38984eb187ff612e73e355fe43a66eff36bba3607c3bd268031 not found: ID does not exist" Nov 02 12:16:40.580281 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:40.580256 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-97f775\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="20b1f1b9-6a37-4e7e-b4ba-9c472d322858" pod="opendatahub/test-curl-7227-97f775" Nov 02 12:16:41.074455 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.074425 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b1f1b9-6a37-4e7e-b4ba-9c472d322858" path="/var/lib/kubelet/pods/20b1f1b9-6a37-4e7e-b4ba-9c472d322858/volumes" Nov 02 12:16:41.158202 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.158177 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7227-634ce0"] Nov 02 12:16:41.162758 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.162732 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7227-634ce0"] Nov 02 12:16:41.308516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.308467 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7227-6891a7"] Nov 02 12:16:41.308900 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.308893 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4d5e66b1-037f-45f2-ba71-e189766cae70" containerName="test-curl-7227-634ce0" Nov 02 12:16:41.308948 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.308905 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5e66b1-037f-45f2-ba71-e189766cae70" containerName="test-curl-7227-634ce0" Nov 02 12:16:41.308987 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.308982 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5e66b1-037f-45f2-ba71-e189766cae70" containerName="test-curl-7227-634ce0" Nov 02 12:16:41.313106 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.313090 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-6891a7" Nov 02 12:16:41.368615 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.368539 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-6891a7"] Nov 02 12:16:41.393826 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.393800 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xwgd\" (UniqueName: \"kubernetes.io/projected/0940cff0-e354-4f86-b707-ab12b7e055ee-kube-api-access-5xwgd\") pod \"test-curl-7227-6891a7\" (UID: \"0940cff0-e354-4f86-b707-ab12b7e055ee\") " pod="opendatahub/test-curl-7227-6891a7" Nov 02 12:16:41.495321 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.495289 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xwgd\" (UniqueName: \"kubernetes.io/projected/0940cff0-e354-4f86-b707-ab12b7e055ee-kube-api-access-5xwgd\") pod \"test-curl-7227-6891a7\" (UID: \"0940cff0-e354-4f86-b707-ab12b7e055ee\") " pod="opendatahub/test-curl-7227-6891a7" Nov 02 12:16:41.525831 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.525795 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7227-634ce0" podUID="4d5e66b1-037f-45f2-ba71-e189766cae70" containerName="test-curl-7227-634ce0" containerID="cri-o://031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21" gracePeriod=2 Nov 02 12:16:41.555834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.555812 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xwgd\" (UniqueName: \"kubernetes.io/projected/0940cff0-e354-4f86-b707-ab12b7e055ee-kube-api-access-5xwgd\") pod \"test-curl-7227-6891a7\" (UID: \"0940cff0-e354-4f86-b707-ab12b7e055ee\") " pod="opendatahub/test-curl-7227-6891a7" Nov 02 12:16:41.576918 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:41.576887 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-634ce0\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="4d5e66b1-037f-45f2-ba71-e189766cae70" pod="opendatahub/test-curl-7227-634ce0" Nov 02 12:16:41.624245 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.624192 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-6891a7" Nov 02 12:16:41.995458 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:41.995432 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-6891a7"] Nov 02 12:16:41.997588 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:16:41.997560 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0940cff0_e354_4f86_b707_ab12b7e055ee.slice/crio-f1dda9f8b7dbdd66baa70dbfd32e230bf4cbc87c8673e0303d90cf8390aa95bd WatchSource:0}: Error finding container f1dda9f8b7dbdd66baa70dbfd32e230bf4cbc87c8673e0303d90cf8390aa95bd: Status 404 returned error can't find the container with id f1dda9f8b7dbdd66baa70dbfd32e230bf4cbc87c8673e0303d90cf8390aa95bd Nov 02 12:16:42.532014 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:42.531970 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-6891a7" event={"ID":"0940cff0-e354-4f86-b707-ab12b7e055ee","Type":"ContainerStarted","Data":"b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d"} Nov 02 12:16:42.532373 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:42.532021 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-6891a7" event={"ID":"0940cff0-e354-4f86-b707-ab12b7e055ee","Type":"ContainerStarted","Data":"f1dda9f8b7dbdd66baa70dbfd32e230bf4cbc87c8673e0303d90cf8390aa95bd"} Nov 02 12:16:42.646410 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:42.646363 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7227-6891a7" podStartSLOduration=1.646349613 podStartE2EDuration="1.646349613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:16:42.644269378 +0000 UTC m=+1219.674505628" watchObservedRunningTime="2026-11-02 12:16:42.646349613 +0000 UTC m=+1219.676585862" Nov 02 12:16:42.695005 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:42.694970 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-634ce0\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="4d5e66b1-037f-45f2-ba71-e189766cae70" pod="opendatahub/test-curl-7227-634ce0" Nov 02 12:16:43.122171 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:43.122134 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-634ce0\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="4d5e66b1-037f-45f2-ba71-e189766cae70" pod="opendatahub/test-curl-7227-634ce0" Nov 02 12:16:43.763839 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:43.763815 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-634ce0" Nov 02 12:16:43.816336 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:43.816283 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5e66b1-037f-45f2-ba71-e189766cae70-kube-api-access-dfg6h\" (UniqueName: \"kubernetes.io/projected/4d5e66b1-037f-45f2-ba71-e189766cae70-kube-api-access-dfg6h\") pod \"4d5e66b1-037f-45f2-ba71-e189766cae70\" (UID: \"4d5e66b1-037f-45f2-ba71-e189766cae70\") " Nov 02 12:16:43.818121 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:43.818098 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5e66b1-037f-45f2-ba71-e189766cae70-kube-api-access-dfg6h" pod "4d5e66b1-037f-45f2-ba71-e189766cae70" (UID: "4d5e66b1-037f-45f2-ba71-e189766cae70"). InnerVolumeSpecName "kube-api-access-dfg6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:16:43.917403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:43.917366 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfg6h\" (UniqueName: \"kubernetes.io/projected/4d5e66b1-037f-45f2-ba71-e189766cae70-kube-api-access-dfg6h\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:16:44.116807 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:44.116713 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7227-6891a7"] Nov 02 12:16:44.119350 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:44.119325 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7227-6891a7"] Nov 02 12:16:44.342265 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:44.342235 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:44.543618 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:44.543585 4312 generic.go:356] "Generic (PLEG): container finished" podID="4d5e66b1-037f-45f2-ba71-e189766cae70" containerID="031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21" exitCode=137 Nov 02 12:16:44.543804 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:44.543628 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-634ce0" Nov 02 12:16:44.543804 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:44.543679 4312 scope.go:122] "RemoveContainer" containerID="031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21" Nov 02 12:16:44.543989 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:44.543960 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7227-6891a7" podUID="0940cff0-e354-4f86-b707-ab12b7e055ee" containerName="test-curl-7227-6891a7" containerID="cri-o://b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d" gracePeriod=2 Nov 02 12:16:44.552070 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:44.552054 4312 scope.go:122] "RemoveContainer" containerID="031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21" Nov 02 12:16:44.552319 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:44.552304 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21\": container with ID starting with 031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21 not found: ID does not exist" containerID="031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21" Nov 02 12:16:44.552377 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:44.552325 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21"} err="failed to get container status \"031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21\": rpc error: code = NotFound desc = could not find container \"031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21\": container with ID starting with 031c3efbfde4299846b06f43d955a7aa2820a20757d1278a87d3d3189277ac21 not found: ID does not exist" Nov 02 12:16:44.594910 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:44.594885 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-6891a7\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="0940cff0-e354-4f86-b707-ab12b7e055ee" pod="opendatahub/test-curl-7227-6891a7" Nov 02 12:16:44.645828 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:44.645802 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-6891a7\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="0940cff0-e354-4f86-b707-ab12b7e055ee" pod="opendatahub/test-curl-7227-6891a7" Nov 02 12:16:44.696576 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:44.696551 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-634ce0\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="4d5e66b1-037f-45f2-ba71-e189766cae70" pod="opendatahub/test-curl-7227-634ce0" Nov 02 12:16:45.078079 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:45.078041 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5e66b1-037f-45f2-ba71-e189766cae70" path="/var/lib/kubelet/pods/4d5e66b1-037f-45f2-ba71-e189766cae70/volumes" Nov 02 12:16:45.779602 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:45.779563 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" podUID="67d52846-b9e0-4b0c-b03e-f2e90b668969" containerName="main" containerID="cri-o://f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b" gracePeriod=45 Nov 02 12:16:46.059832 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.059809 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:16:46.139393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.139364 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d52846-b9e0-4b0c-b03e-f2e90b668969-kube-api-access-5lmqf\" (UniqueName: \"kubernetes.io/projected/67d52846-b9e0-4b0c-b03e-f2e90b668969-kube-api-access-5lmqf\") pod \"67d52846-b9e0-4b0c-b03e-f2e90b668969\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " Nov 02 12:16:46.139784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.139402 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-tmp-dir\") pod \"67d52846-b9e0-4b0c-b03e-f2e90b668969\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " Nov 02 12:16:46.139784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.139445 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-home\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-home\") pod \"67d52846-b9e0-4b0c-b03e-f2e90b668969\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " Nov 02 12:16:46.139784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.139518 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67d52846-b9e0-4b0c-b03e-f2e90b668969-tls-certs\" (UniqueName: \"kubernetes.io/secret/67d52846-b9e0-4b0c-b03e-f2e90b668969-tls-certs\") pod \"67d52846-b9e0-4b0c-b03e-f2e90b668969\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " Nov 02 12:16:46.139784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.139553 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-dshm\") pod \"67d52846-b9e0-4b0c-b03e-f2e90b668969\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " Nov 02 12:16:46.139784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.139723 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-model-cache\") pod \"67d52846-b9e0-4b0c-b03e-f2e90b668969\" (UID: \"67d52846-b9e0-4b0c-b03e-f2e90b668969\") " Nov 02 12:16:46.139784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.139730 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-tmp-dir" pod "67d52846-b9e0-4b0c-b03e-f2e90b668969" (UID: "67d52846-b9e0-4b0c-b03e-f2e90b668969"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:16:46.139784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.139744 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-home" pod "67d52846-b9e0-4b0c-b03e-f2e90b668969" (UID: "67d52846-b9e0-4b0c-b03e-f2e90b668969"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:16:46.140158 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.139982 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-model-cache" pod "67d52846-b9e0-4b0c-b03e-f2e90b668969" (UID: "67d52846-b9e0-4b0c-b03e-f2e90b668969"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:16:46.140218 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.140156 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:16:46.140218 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.140177 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:16:46.140218 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.140192 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:16:46.141816 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.141788 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-dshm" pod "67d52846-b9e0-4b0c-b03e-f2e90b668969" (UID: "67d52846-b9e0-4b0c-b03e-f2e90b668969"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:16:46.141925 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.141851 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d52846-b9e0-4b0c-b03e-f2e90b668969-kube-api-access-5lmqf" pod "67d52846-b9e0-4b0c-b03e-f2e90b668969" (UID: "67d52846-b9e0-4b0c-b03e-f2e90b668969"). InnerVolumeSpecName "kube-api-access-5lmqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:16:46.141925 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.141852 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d52846-b9e0-4b0c-b03e-f2e90b668969-tls-certs" pod "67d52846-b9e0-4b0c-b03e-f2e90b668969" (UID: "67d52846-b9e0-4b0c-b03e-f2e90b668969"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:16:46.241451 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.241420 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lmqf\" (UniqueName: \"kubernetes.io/projected/67d52846-b9e0-4b0c-b03e-f2e90b668969-kube-api-access-5lmqf\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:16:46.241451 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.241446 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67d52846-b9e0-4b0c-b03e-f2e90b668969-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:16:46.241628 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.241459 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67d52846-b9e0-4b0c-b03e-f2e90b668969-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:16:46.555629 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.555595 4312 generic.go:356] "Generic (PLEG): container finished" podID="67d52846-b9e0-4b0c-b03e-f2e90b668969" containerID="f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b" exitCode=0 Nov 02 12:16:46.555749 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.555643 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" Nov 02 12:16:46.555749 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.555675 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" event={"ID":"67d52846-b9e0-4b0c-b03e-f2e90b668969","Type":"ContainerDied","Data":"f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b"} Nov 02 12:16:46.555749 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.555727 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9" event={"ID":"67d52846-b9e0-4b0c-b03e-f2e90b668969","Type":"ContainerDied","Data":"b4b05311d5151e811b3fac2250b84bd9160005143b323c8d6e5a7b739de00283"} Nov 02 12:16:46.555920 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.555752 4312 scope.go:122] "RemoveContainer" containerID="f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b" Nov 02 12:16:46.600279 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.600259 4312 scope.go:122] "RemoveContainer" containerID="f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b" Nov 02 12:16:46.600625 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:46.600602 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b\": container with ID starting with f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b not found: ID does not exist" containerID="f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b" Nov 02 12:16:46.600715 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.600638 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b"} err="failed to get container status \"f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b\": rpc error: code = NotFound desc = could not find container \"f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b\": container with ID starting with f8caf5b19a2d173ae22749e9c04f6a0ecb920730c4ff53f54a5c5eb467f4371b not found: ID does not exist" Nov 02 12:16:46.724691 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.724667 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9"] Nov 02 12:16:46.777395 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.777369 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-03c4e8/auth-test-model-03c4e8-kserve-7c4dc959df-2rch9"] Nov 02 12:16:46.779058 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.779040 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-6891a7" Nov 02 12:16:46.847005 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.846951 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0940cff0-e354-4f86-b707-ab12b7e055ee-kube-api-access-5xwgd\" (UniqueName: \"kubernetes.io/projected/0940cff0-e354-4f86-b707-ab12b7e055ee-kube-api-access-5xwgd\") pod \"0940cff0-e354-4f86-b707-ab12b7e055ee\" (UID: \"0940cff0-e354-4f86-b707-ab12b7e055ee\") " Nov 02 12:16:46.848729 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.848702 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0940cff0-e354-4f86-b707-ab12b7e055ee-kube-api-access-5xwgd" pod "0940cff0-e354-4f86-b707-ab12b7e055ee" (UID: "0940cff0-e354-4f86-b707-ab12b7e055ee"). InnerVolumeSpecName "kube-api-access-5xwgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:16:46.948402 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:46.948376 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xwgd\" (UniqueName: \"kubernetes.io/projected/0940cff0-e354-4f86-b707-ab12b7e055ee-kube-api-access-5xwgd\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:16:47.076661 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:47.076627 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0940cff0-e354-4f86-b707-ab12b7e055ee" path="/var/lib/kubelet/pods/0940cff0-e354-4f86-b707-ab12b7e055ee/volumes" Nov 02 12:16:47.077094 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:47.077075 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d52846-b9e0-4b0c-b03e-f2e90b668969" path="/var/lib/kubelet/pods/67d52846-b9e0-4b0c-b03e-f2e90b668969/volumes" Nov 02 12:16:47.565132 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:47.565091 4312 generic.go:356] "Generic (PLEG): container finished" podID="0940cff0-e354-4f86-b707-ab12b7e055ee" containerID="b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d" exitCode=137 Nov 02 12:16:47.565132 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:47.565111 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-6891a7" Nov 02 12:16:47.565132 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:47.565136 4312 scope.go:122] "RemoveContainer" containerID="b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d" Nov 02 12:16:47.573821 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:47.573796 4312 scope.go:122] "RemoveContainer" containerID="b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d" Nov 02 12:16:47.574128 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:47.574104 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d\": container with ID starting with b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d not found: ID does not exist" containerID="b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d" Nov 02 12:16:47.574176 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:47.574144 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d"} err="failed to get container status \"b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d\": rpc error: code = NotFound desc = could not find container \"b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d\": container with ID starting with b9cc327c4df25791cec9db56d6d09dc9ea40e427856330ed5988daf5c9c60a3d not found: ID does not exist" Nov 02 12:16:49.189897 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:49.189869 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:54.095629 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:54.095578 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:59.196061 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:16:59.196021 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:16:59.372082 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:16:59.372053 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:17:03.727840 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.727808 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb"] Nov 02 12:17:03.728292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.728260 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0940cff0-e354-4f86-b707-ab12b7e055ee" containerName="test-curl-7227-6891a7" Nov 02 12:17:03.728292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.728274 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0940cff0-e354-4f86-b707-ab12b7e055ee" containerName="test-curl-7227-6891a7" Nov 02 12:17:03.728292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.728293 4312 cpu_manager.go:414] "containerMap: removing container" podUID="67d52846-b9e0-4b0c-b03e-f2e90b668969" containerName="main" Nov 02 12:17:03.728392 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.728298 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d52846-b9e0-4b0c-b03e-f2e90b668969" containerName="main" Nov 02 12:17:03.728392 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.728388 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d52846-b9e0-4b0c-b03e-f2e90b668969" containerName="main" Nov 02 12:17:03.728455 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.728403 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0940cff0-e354-4f86-b707-ab12b7e055ee" containerName="test-curl-7227-6891a7" Nov 02 12:17:03.732825 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.732807 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:03.784711 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.784684 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-3a78b4-serving-cert\"" Nov 02 12:17:03.836456 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.836431 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb"] Nov 02 12:17:03.855037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.855008 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0f0a6-22c8-465b-99c9-d6028080ce42-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:03.855142 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.855056 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-metrics-certs\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:03.855142 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.855131 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdszq\" (UniqueName: \"kubernetes.io/projected/6ac0f0a6-22c8-465b-99c9-d6028080ce42-kube-api-access-zdszq\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:03.855265 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.855211 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:03.956526 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.956487 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0f0a6-22c8-465b-99c9-d6028080ce42-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:03.956648 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.956540 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-metrics-certs\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:03.956648 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.956597 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdszq\" (UniqueName: \"kubernetes.io/projected/6ac0f0a6-22c8-465b-99c9-d6028080ce42-kube-api-access-zdszq\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:03.956648 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.956629 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:03.956773 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:17:03.956757 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-serving-cert: secret "maas-api-e2e-extmod-b-3a78b4-serving-cert" not found Nov 02 12:17:03.956837 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:17:03.956826 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-maas-api-tls podName:6ac0f0a6-22c8-465b-99c9-d6028080ce42 nodeName:}" failed. No retries permitted until 2026-11-02 12:17:04.456806247 +0000 UTC m=+1241.487042493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-maas-api-tls") pod "maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" (UID: "6ac0f0a6-22c8-465b-99c9-d6028080ce42") : secret "maas-api-e2e-extmod-b-3a78b4-serving-cert" not found Nov 02 12:17:03.957231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.957211 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0f0a6-22c8-465b-99c9-d6028080ce42-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:03.958919 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:03.958901 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-metrics-certs\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:04.017308 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:04.017236 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdszq\" (UniqueName: \"kubernetes.io/projected/6ac0f0a6-22c8-465b-99c9-d6028080ce42-kube-api-access-zdszq\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:04.025757 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:04.025723 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:04.272955 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:04.272872 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:04.464838 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:04.464792 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:04.467307 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:04.467284 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-maas-api-tls\") pod \"maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:04.520017 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:04.519994 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:04.645647 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:04.645579 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:05.024573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:05.024551 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb"] Nov 02 12:17:05.026694 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:17:05.026667 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac0f0a6_22c8_465b_99c9_d6028080ce42.slice/crio-89fa85be8f6e380b68a0d7ad01e15ed79c3fe47c320c6cf655da461cce7ad3bd WatchSource:0}: Error finding container 89fa85be8f6e380b68a0d7ad01e15ed79c3fe47c320c6cf655da461cce7ad3bd: Status 404 returned error can't find the container with id 89fa85be8f6e380b68a0d7ad01e15ed79c3fe47c320c6cf655da461cce7ad3bd Nov 02 12:17:05.692182 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:05.692144 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" event={"ID":"6ac0f0a6-22c8-465b-99c9-d6028080ce42","Type":"ContainerStarted","Data":"89fa85be8f6e380b68a0d7ad01e15ed79c3fe47c320c6cf655da461cce7ad3bd"} Nov 02 12:17:06.656282 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.656245 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr"] Nov 02 12:17:06.659823 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.659801 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.699270 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.699228 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" event={"ID":"6ac0f0a6-22c8-465b-99c9-d6028080ce42","Type":"ContainerStarted","Data":"02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232"} Nov 02 12:17:06.699389 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.699320 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:06.761292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.761267 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-dd087a\"/\"rate-test-model-dd087a-kserve-self-signed-certs\"" Nov 02 12:17:06.761292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.761289 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-dd087a\"/\"default-dockercfg-s8lc4\"" Nov 02 12:17:06.761545 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.761388 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-dd087a\"/\"kube-root-ca.crt\"" Nov 02 12:17:06.761545 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.761475 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-dd087a\"/\"openshift-service-ca.crt\"" Nov 02 12:17:06.764978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.764953 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr"] Nov 02 12:17:06.788765 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.788740 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-tmp-dir\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.788863 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.788780 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc76abd4-2a9d-4238-a07e-78ceb932d188-tls-certs\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.788905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.788870 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-home\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.788986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.788970 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thwxm\" (UniqueName: \"kubernetes.io/projected/cc76abd4-2a9d-4238-a07e-78ceb932d188-kube-api-access-thwxm\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.789110 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.789092 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-dshm\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.789269 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.789250 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-model-cache\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.852326 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.852269 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:06.890825 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.890796 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thwxm\" (UniqueName: \"kubernetes.io/projected/cc76abd4-2a9d-4238-a07e-78ceb932d188-kube-api-access-thwxm\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.891022 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.890843 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-dshm\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.891022 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.890888 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-model-cache\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.891022 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.890915 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-tmp-dir\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.891022 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.890939 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc76abd4-2a9d-4238-a07e-78ceb932d188-tls-certs\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.891022 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.890982 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-home\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.891325 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.891303 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-model-cache\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.891378 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.891355 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-tmp-dir\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.891414 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.891386 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-home\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.893595 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.893568 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc76abd4-2a9d-4238-a07e-78ceb932d188-tls-certs\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.893715 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.893570 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-dshm\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:06.929927 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.929883 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" podStartSLOduration=3.122415053 podStartE2EDuration="3.929871172s" totalImagesPullingTime="807.456119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:17:06.928129089 +0000 UTC m=+1243.958365341" watchObservedRunningTime="2026-11-02 12:17:06.929871172 +0000 UTC m=+1243.960107422" Nov 02 12:17:06.979938 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:06.979910 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thwxm\" (UniqueName: \"kubernetes.io/projected/cc76abd4-2a9d-4238-a07e-78ceb932d188-kube-api-access-thwxm\") pod \"rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:07.274862 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:07.274783 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:07.662766 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:07.662739 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr"] Nov 02 12:17:07.665198 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:17:07.665175 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc76abd4_2a9d_4238_a07e_78ceb932d188.slice/crio-8bb3fb84db377e83dd211dfd85e8f7dbc35b7cd29e07ad331175ef9959886bb9 WatchSource:0}: Error finding container 8bb3fb84db377e83dd211dfd85e8f7dbc35b7cd29e07ad331175ef9959886bb9: Status 404 returned error can't find the container with id 8bb3fb84db377e83dd211dfd85e8f7dbc35b7cd29e07ad331175ef9959886bb9 Nov 02 12:17:07.706361 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:07.706335 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" event={"ID":"cc76abd4-2a9d-4238-a07e-78ceb932d188","Type":"ContainerStarted","Data":"8bb3fb84db377e83dd211dfd85e8f7dbc35b7cd29e07ad331175ef9959886bb9"} Nov 02 12:17:08.713376 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:08.713339 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" event={"ID":"cc76abd4-2a9d-4238-a07e-78ceb932d188","Type":"ContainerStarted","Data":"5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d"} Nov 02 12:17:08.880119 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:08.880070 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" podStartSLOduration=2.880055675 podStartE2EDuration="2.880055675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:17:08.829401438 +0000 UTC m=+1245.859637688" watchObservedRunningTime="2026-11-02 12:17:08.880055675 +0000 UTC m=+1245.910291925" Nov 02 12:17:09.038865 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:09.038775 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:09.275258 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:09.275226 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:10.466668 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:10.466639 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:12.735332 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:12.735300 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 12:17:14.142248 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:14.142217 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:17.275399 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:17.275361 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:17.288251 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:17.288226 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:17.821684 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:17.821655 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:17:19.112242 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:19.112207 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:24.076661 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:24.076622 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:24.329355 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:24.329274 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:29.230995 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:29.230959 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:29.375288 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:17:29.375245 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:17:34.199571 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:34.199477 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:39.106036 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:39.106006 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:42.921282 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:42.921163 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:44.839437 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:44.839402 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:49.789535 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:49.789481 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:51.424740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:51.424704 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:54.953882 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:54.953841 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:57.733558 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:57.733525 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:58.868655 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:58.868625 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:59.130481 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:59.130385 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:59.372194 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:17:59.372162 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:17:59.391941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:59.391855 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:17:59.807929 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:17:59.807898 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:00.713336 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:00.713308 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:00.985116 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:00.985038 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:05.036039 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:05.036000 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:06.018516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:06.018406 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:10.870486 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:10.870447 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:11.502522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:11.502196 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:16.467227 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:16.467193 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:21.389683 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:21.389641 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:24.756640 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.756600 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn"] Nov 02 12:18:24.760405 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.760383 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.816300 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.816272 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-dshm\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.816451 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.816336 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-home\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.816600 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.816573 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-model-cache\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.816663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.816616 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9tz9\" (UniqueName: \"kubernetes.io/projected/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-kube-api-access-z9tz9\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.816663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.816649 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tmp-dir\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.816794 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.816773 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tls-certs\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.861833 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.861804 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1703c\"/\"distinct-2-w0-b1703c-kserve-self-signed-certs\"" Nov 02 12:18:24.864110 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.864085 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn"] Nov 02 12:18:24.918214 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.918174 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-dshm\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.918409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.918262 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-home\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.918409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.918316 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-model-cache\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.918409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.918344 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9tz9\" (UniqueName: \"kubernetes.io/projected/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-kube-api-access-z9tz9\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.918409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.918376 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tmp-dir\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.918660 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.918527 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tls-certs\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.918837 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.918820 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-home\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.918928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.918843 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-model-cache\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.918928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.918852 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tmp-dir\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.920636 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.920610 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-dshm\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.920964 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.920944 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tls-certs\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:24.952270 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.952243 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:24.979042 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:24.979020 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9tz9\" (UniqueName: \"kubernetes.io/projected/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-kube-api-access-z9tz9\") pod \"distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:25.071466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:25.071391 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:25.463473 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:25.463451 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn"] Nov 02 12:18:25.465005 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:18:25.464980 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdc0f39f_aca8_4e76_a682_f01a1474c9f1.slice/crio-ccf9750b5c798e82025568b2b673fee7498618db8da4b0c6e8bd222ce40c71f9 WatchSource:0}: Error finding container ccf9750b5c798e82025568b2b673fee7498618db8da4b0c6e8bd222ce40c71f9: Status 404 returned error can't find the container with id ccf9750b5c798e82025568b2b673fee7498618db8da4b0c6e8bd222ce40c71f9 Nov 02 12:18:26.317396 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:26.317366 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:26.418296 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:26.418269 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" event={"ID":"cdc0f39f-aca8-4e76-a682-f01a1474c9f1","Type":"ContainerStarted","Data":"bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b"} Nov 02 12:18:26.418437 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:26.418306 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" event={"ID":"cdc0f39f-aca8-4e76-a682-f01a1474c9f1","Type":"ContainerStarted","Data":"ccf9750b5c798e82025568b2b673fee7498618db8da4b0c6e8bd222ce40c71f9"} Nov 02 12:18:26.541251 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:26.541185 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" podStartSLOduration=2.541164064 podStartE2EDuration="2.541164064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:18:26.538121512 +0000 UTC m=+1323.568357775" watchObservedRunningTime="2026-11-02 12:18:26.541164064 +0000 UTC m=+1323.571400315" Nov 02 12:18:27.075598 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:27.075564 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:29.513520 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:18:29.504685 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:18:31.371279 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:31.371234 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:33.268720 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:33.268682 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:33.703160 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:33.703132 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:35.077834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:35.077804 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:35.085933 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:35.085912 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:35.506777 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:35.506748 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:36.292011 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:36.291979 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:39.554875 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:39.554844 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:39.809866 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:39.809780 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:40.162705 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:40.162672 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:40.478697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:40.478600 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:41.918525 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:41.917031 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:42.515199 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:42.513996 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" podUID="f01b2ae0-f424-46de-976f-fd2188cbfb6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:18:42.559523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:42.557451 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" podUID="cdc0f39f-aca8-4e76-a682-f01a1474c9f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 12:18:42.583967 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:42.583928 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:18:42.608591 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:42.608555 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:18:43.587131 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:43.587088 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" podUID="cdc0f39f-aca8-4e76-a682-f01a1474c9f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:18:43.595515 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:43.588065 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:18:43.596703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:43.596218 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:43.596942 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:43.596926 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:18:43.613522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:43.611582 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:18:43.613522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:43.611965 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:18:43.643812 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:43.643782 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:18:43.653932 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:43.653904 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:18:43.702203 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:43.702175 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:18:46.307712 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:46.307679 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:47.492749 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:47.492718 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:52.403465 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:52.403425 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:57.303904 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:18:57.303866 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:18:59.374028 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:18:59.373991 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:19:02.835013 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:02.834967 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" podUID="cc76abd4-2a9d-4238-a07e-78ceb932d188" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:19:04.033914 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:04.033884 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:06.268882 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:06.268849 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:06.523727 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:06.523655 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:08.094475 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:08.094436 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:11.209235 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:11.209205 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:11.473685 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:11.473611 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:12.349256 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:12.349219 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:14.410973 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:14.410938 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:14.689445 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:14.689411 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:19.407699 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:19.407664 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:19.662869 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:19.662840 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:21.112706 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:21.112670 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:21.738492 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:21.738456 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:22.484890 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:22.484854 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:22.874733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:22.874656 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:23.124411 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:23.124383 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:24.690292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:24.690258 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:27.999947 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:27.999874 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:28.256073 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:28.255973 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:29.206402 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:29.206350 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:29.381578 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:19:29.381540 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:19:29.640183 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:29.640109 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:30.863244 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:30.863209 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:31.115656 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:31.115565 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:31.374120 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:31.374020 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:35.361892 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:35.361858 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:36.299006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:36.298968 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:36.545268 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:36.545239 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:37.939907 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:37.939873 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:41.586631 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:41.586595 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:44.622571 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:44.622530 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:44.879924 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:44.879830 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:46.646665 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:46.646621 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:19:46.656903 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:46.656855 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 12:19:46.708016 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:46.707973 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" podUID="cdc0f39f-aca8-4e76-a682-f01a1474c9f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Nov 02 12:19:46.811530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:46.810644 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" podUID="cc76abd4-2a9d-4238-a07e-78ceb932d188" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 02 12:19:46.983641 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:46.983599 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 12:19:47.516652 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:47.516601 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" podUID="f01b2ae0-f424-46de-976f-fd2188cbfb6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:19:47.813702 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:47.813575 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" podUID="cc76abd4-2a9d-4238-a07e-78ceb932d188" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:19:47.814935 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:47.814915 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:19:47.943788 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:47.943742 4312 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:19:48.065887 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:48.065139 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:19:48.069484 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:48.069362 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:48.406486 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:48.406360 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:48.668727 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:48.668702 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:53.995831 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:53.995791 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:54.742483 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:54.742375 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:55.000364 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:55.000238 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:59.131476 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:19:59.131429 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:19:59.372540 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:19:59.372488 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:20:03.657283 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:03.657217 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:20:03.701685 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:03.701624 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 12:20:03.710645 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:03.710571 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" podUID="cdc0f39f-aca8-4e76-a682-f01a1474c9f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:20:04.868899 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:04.868864 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:10.047890 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:10.047854 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm"] Nov 02 12:20:10.097062 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:10.097035 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42cb2ce7-88nlm"] Nov 02 12:20:10.888255 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:10.888220 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:11.075883 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:11.075844 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b649583-6f05-4a86-9e8b-92c650a964c5" path="/var/lib/kubelet/pods/6b649583-6f05-4a86-9e8b-92c650a964c5/volumes" Nov 02 12:20:11.567735 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:11.567704 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:15.807379 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:15.807330 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:17.440806 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:17.440771 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:17.690210 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:17.690170 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:19.026705 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:19.026670 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:19.397726 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:19.397652 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw"] Nov 02 12:20:19.397884 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:19.397861 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" containerID="cri-o://bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab" gracePeriod=30 Nov 02 12:20:19.500379 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:19.500351 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:20.816738 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:20.816701 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:22.090617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:22.090583 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:22.395227 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:22.395149 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:22.784207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:22.784172 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:24.658743 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.658717 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:20:24.803715 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.803650 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-socket\") pod \"e8106eef-1121-4fe1-aba8-99b6f485bcee\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " Nov 02 12:20:24.803715 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.803701 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-podinfo\") pod \"e8106eef-1121-4fe1-aba8-99b6f485bcee\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " Nov 02 12:20:24.803918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.803720 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-kube-api-access-mnx6c\" (UniqueName: \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-kube-api-access-mnx6c\") pod \"e8106eef-1121-4fe1-aba8-99b6f485bcee\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " Nov 02 12:20:24.803918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.803749 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-envoy\") pod \"e8106eef-1121-4fe1-aba8-99b6f485bcee\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " Nov 02 12:20:24.803918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.803805 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-credential-socket\") pod \"e8106eef-1121-4fe1-aba8-99b6f485bcee\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " Nov 02 12:20:24.803918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.803865 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-certs\") pod \"e8106eef-1121-4fe1-aba8-99b6f485bcee\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " Nov 02 12:20:24.803918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.803904 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-data\") pod \"e8106eef-1121-4fe1-aba8-99b6f485bcee\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " Nov 02 12:20:24.804179 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.803944 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-token\" (UniqueName: \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-token\") pod \"e8106eef-1121-4fe1-aba8-99b6f485bcee\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " Nov 02 12:20:24.804179 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.803968 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8106eef-1121-4fe1-aba8-99b6f485bcee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8106eef-1121-4fe1-aba8-99b6f485bcee-istiod-ca-cert\") pod \"e8106eef-1121-4fe1-aba8-99b6f485bcee\" (UID: \"e8106eef-1121-4fe1-aba8-99b6f485bcee\") " Nov 02 12:20:24.804579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.803958 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-socket" pod "e8106eef-1121-4fe1-aba8-99b6f485bcee" (UID: "e8106eef-1121-4fe1-aba8-99b6f485bcee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:20:24.804579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.804171 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-certs" pod "e8106eef-1121-4fe1-aba8-99b6f485bcee" (UID: "e8106eef-1121-4fe1-aba8-99b6f485bcee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:20:24.804579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.804333 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-credential-socket" pod "e8106eef-1121-4fe1-aba8-99b6f485bcee" (UID: "e8106eef-1121-4fe1-aba8-99b6f485bcee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:20:24.804579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.804542 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-data" pod "e8106eef-1121-4fe1-aba8-99b6f485bcee" (UID: "e8106eef-1121-4fe1-aba8-99b6f485bcee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:20:24.805240 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.805213 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8106eef-1121-4fe1-aba8-99b6f485bcee-istiod-ca-cert" pod "e8106eef-1121-4fe1-aba8-99b6f485bcee" (UID: "e8106eef-1121-4fe1-aba8-99b6f485bcee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:20:24.806374 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.806350 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-token" pod "e8106eef-1121-4fe1-aba8-99b6f485bcee" (UID: "e8106eef-1121-4fe1-aba8-99b6f485bcee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:20:24.806481 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.806436 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-kube-api-access-mnx6c" pod "e8106eef-1121-4fe1-aba8-99b6f485bcee" (UID: "e8106eef-1121-4fe1-aba8-99b6f485bcee"). InnerVolumeSpecName "kube-api-access-mnx6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:20:24.806657 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.806536 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-podinfo" pod "e8106eef-1121-4fe1-aba8-99b6f485bcee" (UID: "e8106eef-1121-4fe1-aba8-99b6f485bcee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:20:24.806741 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.806719 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-envoy" pod "e8106eef-1121-4fe1-aba8-99b6f485bcee" (UID: "e8106eef-1121-4fe1-aba8-99b6f485bcee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:20:24.905131 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.905099 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8106eef-1121-4fe1-aba8-99b6f485bcee-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:20:24.905131 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.905127 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:20:24.905281 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.905138 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:20:24.905281 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.905147 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnx6c\" (UniqueName: \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-kube-api-access-mnx6c\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:20:24.905281 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.905157 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:20:24.905281 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.905166 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:20:24.905281 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.905174 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:20:24.905281 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.905182 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:20:24.905281 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:24.905196 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8106eef-1121-4fe1-aba8-99b6f485bcee-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:20:25.366861 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.366782 4312 generic.go:356] "Generic (PLEG): container finished" podID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerID="bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab" exitCode=0 Nov 02 12:20:25.366861 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.366832 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" Nov 02 12:20:25.367068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.366831 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" event={"ID":"e8106eef-1121-4fe1-aba8-99b6f485bcee","Type":"ContainerDied","Data":"bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab"} Nov 02 12:20:25.367068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.366955 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw" event={"ID":"e8106eef-1121-4fe1-aba8-99b6f485bcee","Type":"ContainerDied","Data":"9bafbec8f79a0b9da456e84951a4c2f5ff64d4c41d343afb2accd3bddc1b9a62"} Nov 02 12:20:25.367068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.366984 4312 scope.go:122] "RemoveContainer" containerID="bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab" Nov 02 12:20:25.376274 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.376256 4312 scope.go:122] "RemoveContainer" containerID="c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b" Nov 02 12:20:25.384538 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.384522 4312 scope.go:122] "RemoveContainer" containerID="bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab" Nov 02 12:20:25.384803 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:20:25.384784 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab\": container with ID starting with bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab not found: ID does not exist" containerID="bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab" Nov 02 12:20:25.384870 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.384812 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab"} err="failed to get container status \"bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab\": rpc error: code = NotFound desc = could not find container \"bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab\": container with ID starting with bd3655038f64aedba19dabe6bbe9b35caceb9539a72a630ff6fbfea568a543ab not found: ID does not exist" Nov 02 12:20:25.384870 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.384830 4312 scope.go:122] "RemoveContainer" containerID="c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b" Nov 02 12:20:25.385080 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:20:25.385061 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b\": container with ID starting with c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b not found: ID does not exist" containerID="c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b" Nov 02 12:20:25.385124 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.385087 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b"} err="failed to get container status \"c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b\": rpc error: code = NotFound desc = could not find container \"c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b\": container with ID starting with c9ad55282edc1f7b2122f8d8c0847f9cdc31d118a5912a01d7b99f7128a6fd1b not found: ID does not exist" Nov 02 12:20:25.581875 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.581847 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw"] Nov 02 12:20:25.630757 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:25.630695 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-036f88-openshift-default-dbc8cc695-vm5lw"] Nov 02 12:20:26.042068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:26.042030 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:26.289720 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:26.289690 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:27.076370 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:27.076337 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" path="/var/lib/kubelet/pods/e8106eef-1121-4fe1-aba8-99b6f485bcee/volumes" Nov 02 12:20:29.132273 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:29.132236 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:29.419177 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:29.419146 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:29.436207 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:20:29.436180 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:20:29.686441 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:29.686328 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:30.112463 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:30.112368 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:30.725706 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:30.725675 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:31.508393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:31.508345 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:34.640217 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:34.640170 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:34.892719 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:34.892638 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:36.840926 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:36.840889 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:37.880814 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:37.880777 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:38.129781 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:38.129745 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:42.003667 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:42.003630 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:46.811639 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:46.811603 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:51.792088 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:51.792056 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:56.691485 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:20:56.691451 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:20:59.439772 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:20:59.439728 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:21:01.695429 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:01.695396 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:06.601949 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:06.601867 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:11.507756 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:11.507713 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:11.894532 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:11.894425 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:14.570039 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:14.570004 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:16.703056 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:16.703018 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:16.987723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:16.987641 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:17.252113 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:17.252030 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:18.290007 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.289976 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm"] Nov 02 12:21:18.290405 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.290393 4312 cpu_manager.go:414] "containerMap: removing container" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" Nov 02 12:21:18.290453 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.290406 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" Nov 02 12:21:18.290453 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.290414 4312 cpu_manager.go:414] "containerMap: removing container" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" Nov 02 12:21:18.290453 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.290419 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" Nov 02 12:21:18.290581 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.290484 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" Nov 02 12:21:18.290581 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.290505 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8106eef-1121-4fe1-aba8-99b6f485bcee" containerName="istio-proxy" Nov 02 12:21:18.293418 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.293403 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.351303 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.351281 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6a432197\"/\"kube-root-ca.crt\"" Nov 02 12:21:18.354850 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.354826 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b31d6ac5-c822-4337-ab22-a768627bb288-tls-certs\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.354971 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.354875 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mjgh\" (UniqueName: \"kubernetes.io/projected/b31d6ac5-c822-4337-ab22-a768627bb288-kube-api-access-6mjgh\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.355038 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.355009 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-tmp-dir\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.355101 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.355044 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-model-cache\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.355157 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.355145 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-dshm\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.355260 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.355246 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-home\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.362313 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.362292 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6a432197\"/\"openshift-service-ca.crt\"" Nov 02 12:21:18.395217 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.395189 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6a432197\"/\"e2e-lifecycle-model-6a432197-kserve-self-signed-certs\"" Nov 02 12:21:18.395319 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.395268 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6a432197\"/\"default-dockercfg-nxt85\"" Nov 02 12:21:18.397485 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.397466 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm"] Nov 02 12:21:18.456376 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.456347 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-home\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.456602 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.456391 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b31d6ac5-c822-4337-ab22-a768627bb288-tls-certs\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.456602 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.456530 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjgh\" (UniqueName: \"kubernetes.io/projected/b31d6ac5-c822-4337-ab22-a768627bb288-kube-api-access-6mjgh\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.456602 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.456595 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-tmp-dir\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.456785 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.456621 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-model-cache\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.456785 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.456728 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-dshm\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.456878 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.456821 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-home\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.456935 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.456900 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-tmp-dir\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.457046 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.456996 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-model-cache\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.458818 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.458797 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-dshm\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.458983 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.458966 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b31d6ac5-c822-4337-ab22-a768627bb288-tls-certs\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.517143 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.517108 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mjgh\" (UniqueName: \"kubernetes.io/projected/b31d6ac5-c822-4337-ab22-a768627bb288-kube-api-access-6mjgh\") pod \"e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.564987 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.564905 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:18.604877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.604848 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:18.987427 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:18.987395 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm"] Nov 02 12:21:18.989312 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:21:18.989283 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb31d6ac5_c822_4337_ab22_a768627bb288.slice/crio-ef28e2d13de89857b3c35f712b48303caadd161c70dc00c7109ab897159a3332 WatchSource:0}: Error finding container ef28e2d13de89857b3c35f712b48303caadd161c70dc00c7109ab897159a3332: Status 404 returned error can't find the container with id ef28e2d13de89857b3c35f712b48303caadd161c70dc00c7109ab897159a3332 Nov 02 12:21:19.785730 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:19.785695 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" event={"ID":"b31d6ac5-c822-4337-ab22-a768627bb288","Type":"ContainerStarted","Data":"c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf"} Nov 02 12:21:19.785730 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:19.785737 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" event={"ID":"b31d6ac5-c822-4337-ab22-a768627bb288","Type":"ContainerStarted","Data":"ef28e2d13de89857b3c35f712b48303caadd161c70dc00c7109ab897159a3332"} Nov 02 12:21:19.903059 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:19.903004 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" podStartSLOduration=1.90298755 podStartE2EDuration="1.90298755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:21:19.901918826 +0000 UTC m=+1496.932155104" watchObservedRunningTime="2026-11-02 12:21:19.90298755 +0000 UTC m=+1496.933223801" Nov 02 12:21:20.605999 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:20.605967 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:21.999782 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:21.999741 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:23.280563 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:23.280531 4312 scope.go:122] "RemoveContainer" containerID="b34b93387d3f7ae562d5720fe3a259113ae648a0cfae3d45f395a39c6e57c2ea" Nov 02 12:21:26.882666 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:26.882629 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:27.129857 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:27.129800 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:28.605669 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:28.605634 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:28.619531 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:28.619491 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:28.847104 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:28.847073 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:21:29.377031 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:21:29.376998 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:21:32.172265 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:32.172228 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:37.097987 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:37.097949 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:42.113195 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:42.113120 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:47.044469 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:47.044436 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:51.820968 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:51.820937 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:56.753988 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:21:56.753956 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:21:59.371829 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:21:59.371794 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:22:01.695999 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:01.695967 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:06.634089 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:06.634054 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:07.957530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:07.956581 4312 patch_prober.go:29] interesting pod/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 12:22:07.957530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:07.956643 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 12:22:11.993770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:11.993721 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:12.242058 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:12.242013 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:12.486687 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:12.486655 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:17.238225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:17.238188 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:22.489190 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:22.489137 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:24.465007 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:24.464976 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:24.712703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:24.712653 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:27.505754 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:27.505719 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:29.376258 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:22:29.376224 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:22:32.410587 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:32.410555 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:37.327358 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:37.327263 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:42.411855 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:42.411814 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:42.859685 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:42.859592 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:43.149393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:43.149289 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:43.703485 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:43.703444 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:43.949430 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:43.949398 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:46.691670 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:46.691633 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:47.304470 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:47.304440 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:48.254847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:48.254807 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:48.515938 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:48.515853 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:48.760595 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:48.760561 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:49.013797 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:49.013756 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:49.637614 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:49.637586 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:49.993003 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:49.992970 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:50.105601 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.105568 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z"] Nov 02 12:22:50.109204 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.109188 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.164420 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.164392 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-dd087a-openshift-default-dockercfg-7896p\"" Nov 02 12:22:50.217745 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.217675 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z"] Nov 02 12:22:50.242828 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.242793 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.243181 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.243125 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.243339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.243323 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.243461 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.243446 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhh6g\" (UniqueName: \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-kube-api-access-lhh6g\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.243705 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.243687 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb392c56-19f5-4737-8846-7bf9b0911f1e-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.243844 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.243829 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.244159 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.244135 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.244326 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.244281 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.244453 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.244338 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.245976 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.245937 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:50.345206 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345164 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.345393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345238 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.345393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345283 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh6g\" (UniqueName: \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-kube-api-access-lhh6g\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.345393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345359 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb392c56-19f5-4737-8846-7bf9b0911f1e-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.345634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345404 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.345634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345430 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.345634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345460 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.345634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345545 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.345634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345593 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.345925 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345636 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.345925 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345691 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.346037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.345963 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.346037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.346028 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.346359 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.346337 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb392c56-19f5-4737-8846-7bf9b0911f1e-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.347878 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.347856 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.348229 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.348205 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.408320 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.408255 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.408552 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.408523 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhh6g\" (UniqueName: \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-kube-api-access-lhh6g\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.422210 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.422178 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:50.853106 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.853075 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z"] Nov 02 12:22:50.853827 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:22:50.853799 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb392c56_19f5_4737_8846_7bf9b0911f1e.slice/crio-64ffa716d0a9680e0832cdddad6f336b4ace1d4ee2cf90ef763fb2715628dbde WatchSource:0}: Error finding container 64ffa716d0a9680e0832cdddad6f336b4ace1d4ee2cf90ef763fb2715628dbde: Status 404 returned error can't find the container with id 64ffa716d0a9680e0832cdddad6f336b4ace1d4ee2cf90ef763fb2715628dbde Nov 02 12:22:50.856049 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.856018 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:22:50.856160 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.856083 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:22:50.856160 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:50.856116 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:22:51.107344 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:51.107231 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:51.281373 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:51.281331 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" event={"ID":"eb392c56-19f5-4737-8846-7bf9b0911f1e","Type":"ContainerStarted","Data":"4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6"} Nov 02 12:22:51.281554 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:51.281412 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" event={"ID":"eb392c56-19f5-4737-8846-7bf9b0911f1e","Type":"ContainerStarted","Data":"64ffa716d0a9680e0832cdddad6f336b4ace1d4ee2cf90ef763fb2715628dbde"} Nov 02 12:22:51.402948 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:51.402871 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:51.404141 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:51.404091 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" podStartSLOduration=1.404073888 podStartE2EDuration="1.404073888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:22:51.40072195 +0000 UTC m=+1588.430958201" watchObservedRunningTime="2026-11-02 12:22:51.404073888 +0000 UTC m=+1588.434310141" Nov 02 12:22:51.423423 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:51.423394 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:51.425285 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:51.425254 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 12:22:51.425428 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:51.425311 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 12:22:52.423251 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:52.423219 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 12:22:52.423666 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:52.423285 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 12:22:53.423100 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:53.423062 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 12:22:53.423481 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:53.423451 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 12:22:54.156015 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:54.155924 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:54.425402 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:54.425281 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 12:22:54.426185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:54.426023 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 12:22:54.788573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:54.786086 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Nov 02 12:22:54.799521 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:54.794616 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 12:22:54.846710 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:54.845948 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" podUID="b31d6ac5-c822-4337-ab22-a768627bb288" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:22:54.860291 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:54.859566 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" podUID="cdc0f39f-aca8-4e76-a682-f01a1474c9f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 12:22:55.039576 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:55.039069 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:55.076537 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:55.075584 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" podUID="cc76abd4-2a9d-4238-a07e-78ceb932d188" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 02 12:22:55.296637 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:55.296464 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:55.423318 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:55.423281 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 12:22:55.423487 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:55.423339 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 12:22:56.423379 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:56.423341 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 12:22:56.423754 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:56.423408 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 12:22:57.423034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:57.423003 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 12:22:57.423223 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:57.423054 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 12:22:57.736208 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:57.736039 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:57.991939 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:57.991861 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:22:58.624736 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:58.624710 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:58.625133 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:58.625108 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:58.625827 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:58.625806 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:22:59.376126 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:22:59.376094 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:22:59.805838 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:59.805803 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr"] Nov 02 12:22:59.916464 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:22:59.916423 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:00.164005 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:00.163977 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:04.789349 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:04.789309 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:05.445804 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:05.445772 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:05.692179 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:05.692147 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:09.701889 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:09.701856 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:12.593242 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:12.593206 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:14.601132 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:14.601094 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:14.825913 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:14.825878 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" podUID="cc76abd4-2a9d-4238-a07e-78ceb932d188" containerName="main" containerID="cri-o://5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d" gracePeriod=45 Nov 02 12:23:15.106539 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.106483 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:23:15.139545 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.139523 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc76abd4-2a9d-4238-a07e-78ceb932d188-kube-api-access-thwxm\" (UniqueName: \"kubernetes.io/projected/cc76abd4-2a9d-4238-a07e-78ceb932d188-kube-api-access-thwxm\") pod \"cc76abd4-2a9d-4238-a07e-78ceb932d188\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " Nov 02 12:23:15.139658 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.139603 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-home\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-home\") pod \"cc76abd4-2a9d-4238-a07e-78ceb932d188\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " Nov 02 12:23:15.139700 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.139654 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-dshm\") pod \"cc76abd4-2a9d-4238-a07e-78ceb932d188\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " Nov 02 12:23:15.139733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.139705 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-model-cache\") pod \"cc76abd4-2a9d-4238-a07e-78ceb932d188\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " Nov 02 12:23:15.139783 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.139768 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc76abd4-2a9d-4238-a07e-78ceb932d188-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc76abd4-2a9d-4238-a07e-78ceb932d188-tls-certs\") pod \"cc76abd4-2a9d-4238-a07e-78ceb932d188\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " Nov 02 12:23:15.139857 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.139842 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-tmp-dir\") pod \"cc76abd4-2a9d-4238-a07e-78ceb932d188\" (UID: \"cc76abd4-2a9d-4238-a07e-78ceb932d188\") " Nov 02 12:23:15.139918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.139878 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-home" pod "cc76abd4-2a9d-4238-a07e-78ceb932d188" (UID: "cc76abd4-2a9d-4238-a07e-78ceb932d188"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:23:15.139977 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.139924 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-model-cache" pod "cc76abd4-2a9d-4238-a07e-78ceb932d188" (UID: "cc76abd4-2a9d-4238-a07e-78ceb932d188"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:23:15.140109 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.140085 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-tmp-dir" pod "cc76abd4-2a9d-4238-a07e-78ceb932d188" (UID: "cc76abd4-2a9d-4238-a07e-78ceb932d188"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:23:15.140273 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.140255 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:23:15.140395 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.140276 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:23:15.140395 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.140291 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:23:15.141737 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.141714 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-dshm" pod "cc76abd4-2a9d-4238-a07e-78ceb932d188" (UID: "cc76abd4-2a9d-4238-a07e-78ceb932d188"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:23:15.141828 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.141771 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc76abd4-2a9d-4238-a07e-78ceb932d188-tls-certs" pod "cc76abd4-2a9d-4238-a07e-78ceb932d188" (UID: "cc76abd4-2a9d-4238-a07e-78ceb932d188"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:23:15.141875 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.141820 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc76abd4-2a9d-4238-a07e-78ceb932d188-kube-api-access-thwxm" pod "cc76abd4-2a9d-4238-a07e-78ceb932d188" (UID: "cc76abd4-2a9d-4238-a07e-78ceb932d188"). InnerVolumeSpecName "kube-api-access-thwxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:23:15.241573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.241478 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc76abd4-2a9d-4238-a07e-78ceb932d188-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:23:15.241573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.241527 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc76abd4-2a9d-4238-a07e-78ceb932d188-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:23:15.241573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.241546 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thwxm\" (UniqueName: \"kubernetes.io/projected/cc76abd4-2a9d-4238-a07e-78ceb932d188-kube-api-access-thwxm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:23:15.621360 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.621280 4312 generic.go:356] "Generic (PLEG): container finished" podID="cc76abd4-2a9d-4238-a07e-78ceb932d188" containerID="5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d" exitCode=0 Nov 02 12:23:15.621360 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.621323 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" Nov 02 12:23:15.621842 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.621372 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" event={"ID":"cc76abd4-2a9d-4238-a07e-78ceb932d188","Type":"ContainerDied","Data":"5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d"} Nov 02 12:23:15.621842 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.621412 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" event={"ID":"cc76abd4-2a9d-4238-a07e-78ceb932d188","Type":"ContainerDied","Data":"8bb3fb84db377e83dd211dfd85e8f7dbc35b7cd29e07ad331175ef9959886bb9"} Nov 02 12:23:15.621842 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.621435 4312 scope.go:122] "RemoveContainer" containerID="5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d" Nov 02 12:23:15.631633 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.631585 4312 scope.go:122] "RemoveContainer" containerID="5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d" Nov 02 12:23:15.631929 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:23:15.631911 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d\": container with ID starting with 5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d not found: ID does not exist" containerID="5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d" Nov 02 12:23:15.631982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.631939 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d"} err="failed to get container status \"5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d\": rpc error: code = NotFound desc = could not find container \"5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d\": container with ID starting with 5852cb094ed2ce9c7a5a45a37043262580a2e843d653524cb594e7f622662f6d not found: ID does not exist" Nov 02 12:23:15.787807 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.787772 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr"] Nov 02 12:23:15.792788 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:15.792763 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr"] Nov 02 12:23:16.066529 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:16.066477 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-dd087a/rate-test-model-dd087a-kserve-5bc6f94cd9-v4kqr" podUID="cc76abd4-2a9d-4238-a07e-78ceb932d188" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 02 12:23:17.075399 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:17.075367 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc76abd4-2a9d-4238-a07e-78ceb932d188" path="/var/lib/kubelet/pods/cc76abd4-2a9d-4238-a07e-78ceb932d188/volumes" Nov 02 12:23:19.618133 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:19.618099 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:20.363093 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.363060 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz"] Nov 02 12:23:20.363533 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.363518 4312 cpu_manager.go:414] "containerMap: removing container" podUID="cc76abd4-2a9d-4238-a07e-78ceb932d188" containerName="main" Nov 02 12:23:20.363593 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.363534 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc76abd4-2a9d-4238-a07e-78ceb932d188" containerName="main" Nov 02 12:23:20.363632 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.363616 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc76abd4-2a9d-4238-a07e-78ceb932d188" containerName="main" Nov 02 12:23:20.368129 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.368114 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.397257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.397234 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.397383 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.397310 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.397449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.397413 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.397559 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.397537 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.397631 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.397618 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.397680 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.397644 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.397731 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.397704 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sw9p\" (UniqueName: \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-kube-api-access-6sw9p\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.397799 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.397779 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.397851 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.397821 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83952ed-e3f8-42e2-b779-cec3070a5e02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.472467 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.472436 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz"] Nov 02 12:23:20.498982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.498952 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499121 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.498994 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499121 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499023 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sw9p\" (UniqueName: \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-kube-api-access-6sw9p\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499121 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499050 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499121 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499076 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83952ed-e3f8-42e2-b779-cec3070a5e02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499149 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499234 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499271 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499317 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499560 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499385 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499633 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499609 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499705 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499683 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499816 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.499969 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.499951 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83952ed-e3f8-42e2-b779-cec3070a5e02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.501597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.501577 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.501756 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.501734 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.559720 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.559700 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sw9p\" (UniqueName: \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-kube-api-access-6sw9p\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.594028 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.594007 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-lvhwz\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:20.680412 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:20.680390 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:21.125598 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:21.125575 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz"] Nov 02 12:23:21.127339 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:23:21.127300 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83952ed_e3f8_42e2_b779_cec3070a5e02.slice/crio-587aeb86268070634f3fa77b1adb99859d8382077e1121bc435f69acec47ff1f WatchSource:0}: Error finding container 587aeb86268070634f3fa77b1adb99859d8382077e1121bc435f69acec47ff1f: Status 404 returned error can't find the container with id 587aeb86268070634f3fa77b1adb99859d8382077e1121bc435f69acec47ff1f Nov 02 12:23:21.129434 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:21.129405 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:23:21.129533 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:21.129472 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:23:21.129533 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:21.129517 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:23:21.668977 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:21.668929 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" event={"ID":"a83952ed-e3f8-42e2-b779-cec3070a5e02","Type":"ContainerStarted","Data":"515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed"} Nov 02 12:23:21.669200 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:21.669040 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" event={"ID":"a83952ed-e3f8-42e2-b779-cec3070a5e02","Type":"ContainerStarted","Data":"587aeb86268070634f3fa77b1adb99859d8382077e1121bc435f69acec47ff1f"} Nov 02 12:23:21.683720 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:21.683681 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:21.842697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:21.842643 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podStartSLOduration=1.842627834 podStartE2EDuration="1.842627834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:23:21.840342087 +0000 UTC m=+1618.870578339" watchObservedRunningTime="2026-11-02 12:23:21.842627834 +0000 UTC m=+1618.872864083" Nov 02 12:23:22.683645 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:22.683606 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lvhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 12:23:22.683830 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:22.683674 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 12:23:23.681336 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:23.681307 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lvhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 12:23:23.681529 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:23.681354 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 12:23:24.121059 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:24.120975 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:24.644578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:24.644548 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:24.681249 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:24.681220 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lvhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 12:23:24.681480 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:24.681459 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 12:23:25.681141 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:25.681112 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lvhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 12:23:25.681618 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:25.681587 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 12:23:26.680960 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:26.680923 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lvhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 12:23:26.681149 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:26.680998 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 12:23:27.681245 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:27.681210 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lvhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 12:23:27.681748 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:27.681269 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 12:23:28.681336 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:28.681301 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lvhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 02 12:23:28.681855 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:28.681373 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 02 12:23:29.221229 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.221192 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz"] Nov 02 12:23:29.224377 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.224361 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.276413 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.276375 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1cc3a7b3-serving-cert\"" Nov 02 12:23:29.322178 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.322149 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-maas-api-tls\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.322323 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.322198 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-metrics-certs\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.322403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.322382 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsfwk\" (UniqueName: \"kubernetes.io/projected/fc5c069f-5122-4bf3-9301-0df41178c0d6-kube-api-access-rsfwk\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.322454 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.322421 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c069f-5122-4bf3-9301-0df41178c0d6-openshift-service-ca\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.329986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.329961 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz"] Nov 02 12:23:29.403994 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:23:29.403954 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:23:29.423983 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.423942 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-maas-api-tls\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.423983 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.423979 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-metrics-certs\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.424237 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.424034 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsfwk\" (UniqueName: \"kubernetes.io/projected/fc5c069f-5122-4bf3-9301-0df41178c0d6-kube-api-access-rsfwk\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.424237 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.424052 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c069f-5122-4bf3-9301-0df41178c0d6-openshift-service-ca\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.424237 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:23:29.424102 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-serving-cert: secret "maas-api-e2e-migrate-1cc3a7b3-serving-cert" not found Nov 02 12:23:29.424237 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:23:29.424177 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-maas-api-tls podName:fc5c069f-5122-4bf3-9301-0df41178c0d6 nodeName:}" failed. No retries permitted until 2026-11-02 12:23:29.924156016 +0000 UTC m=+1626.954392262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-maas-api-tls") pod "maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" (UID: "fc5c069f-5122-4bf3-9301-0df41178c0d6") : secret "maas-api-e2e-migrate-1cc3a7b3-serving-cert" not found Nov 02 12:23:29.424740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.424721 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c069f-5122-4bf3-9301-0df41178c0d6-openshift-service-ca\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.426810 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.426779 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-metrics-certs\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.507988 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.507771 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsfwk\" (UniqueName: \"kubernetes.io/projected/fc5c069f-5122-4bf3-9301-0df41178c0d6-kube-api-access-rsfwk\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.528457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.528415 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:29.776605 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.776472 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:29.930246 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.930216 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-maas-api-tls\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:29.932596 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:29.932565 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-maas-api-tls\") pod \"maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:30.066140 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:30.066037 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:30.136582 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:30.136547 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:30.523539 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:30.523508 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz"] Nov 02 12:23:30.524311 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:23:30.524280 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5c069f_5122_4bf3_9301_0df41178c0d6.slice/crio-c1d1df6d7ccc89124724b7986ba992e593ffb61c7c6aaf1a57863ebe1d7b12b6 WatchSource:0}: Error finding container c1d1df6d7ccc89124724b7986ba992e593ffb61c7c6aaf1a57863ebe1d7b12b6: Status 404 returned error can't find the container with id c1d1df6d7ccc89124724b7986ba992e593ffb61c7c6aaf1a57863ebe1d7b12b6 Nov 02 12:23:30.525935 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:30.525912 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 12:23:30.725641 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:30.725602 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lvhwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 12:23:30.725817 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:30.725650 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 12:23:30.729946 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:30.729920 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" event={"ID":"fc5c069f-5122-4bf3-9301-0df41178c0d6","Type":"ContainerStarted","Data":"c1d1df6d7ccc89124724b7986ba992e593ffb61c7c6aaf1a57863ebe1d7b12b6"} Nov 02 12:23:30.777697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:30.777618 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:30.778222 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:30.778001 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:30.778820 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:30.778797 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:23:31.737981 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:31.737895 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" event={"ID":"fc5c069f-5122-4bf3-9301-0df41178c0d6","Type":"ContainerStarted","Data":"59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063"} Nov 02 12:23:31.738150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:31.738125 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:31.857212 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:31.857162 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" podStartSLOduration=1.910361649 podStartE2EDuration="2.857146998s" totalImagesPullingTime="946.785349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:23:31.854784492 +0000 UTC m=+1628.885020743" watchObservedRunningTime="2026-11-02 12:23:31.857146998 +0000 UTC m=+1628.887383285" Nov 02 12:23:34.500107 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:34.500072 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:35.477816 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.477787 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45"] Nov 02 12:23:35.481406 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.481390 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.492292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.492267 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.492391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.492310 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.492391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.492338 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a75c277-13b3-417e-a5b3-134cae493061-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.492473 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.492418 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.492473 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.492456 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.492571 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.492477 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.492571 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.492559 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.492641 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.492578 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrh22\" (UniqueName: \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-kube-api-access-jrh22\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.492678 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.492661 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a75c277-13b3-417e-a5b3-134cae493061-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.593855 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.593826 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.593891 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.593926 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a75c277-13b3-417e-a5b3-134cae493061-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.593972 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.594207 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.594256 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.594260 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.594283 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.594311 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrh22\" (UniqueName: \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-kube-api-access-jrh22\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.594351 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.594383 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a75c277-13b3-417e-a5b3-134cae493061-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.594389 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.594703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.594447 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.595047 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.595025 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a75c277-13b3-417e-a5b3-134cae493061-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.596884 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.596860 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.597358 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.597340 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a75c277-13b3-417e-a5b3-134cae493061-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.655557 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.655526 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.655842 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.655826 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrh22\" (UniqueName: \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-kube-api-access-jrh22\") pod \"maas-default-gateway-openshift-default-6dcf757899-mxd45\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:35.684522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.684483 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45"] Nov 02 12:23:35.793824 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:35.793761 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:36.228088 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:36.228064 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45"] Nov 02 12:23:36.229841 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:23:36.229812 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a75c277_13b3_417e_a5b3_134cae493061.slice/crio-26ed42da6229353de2681c9f1083fb4fb3a6de57e5d6e717f779b1cf6b3638a9 WatchSource:0}: Error finding container 26ed42da6229353de2681c9f1083fb4fb3a6de57e5d6e717f779b1cf6b3638a9: Status 404 returned error can't find the container with id 26ed42da6229353de2681c9f1083fb4fb3a6de57e5d6e717f779b1cf6b3638a9 Nov 02 12:23:36.232608 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:36.232572 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:23:36.232684 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:36.232653 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:23:36.232732 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:36.232698 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:23:36.776519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:36.776183 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" event={"ID":"8a75c277-13b3-417e-a5b3-134cae493061","Type":"ContainerStarted","Data":"d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6"} Nov 02 12:23:36.776519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:36.776256 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" event={"ID":"8a75c277-13b3-417e-a5b3-134cae493061","Type":"ContainerStarted","Data":"26ed42da6229353de2681c9f1083fb4fb3a6de57e5d6e717f779b1cf6b3638a9"} Nov 02 12:23:36.794313 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:36.794287 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:36.974017 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:36.964965 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" podStartSLOduration=1.964943569 podStartE2EDuration="1.964943569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:23:36.961977309 +0000 UTC m=+1633.992213560" watchObservedRunningTime="2026-11-02 12:23:36.964943569 +0000 UTC m=+1633.995179821" Nov 02 12:23:37.773477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:37.772844 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:23:37.795749 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:37.795721 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 12:23:37.796096 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:37.795773 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 12:23:38.794125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:38.794091 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 12:23:38.794309 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:38.794141 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 12:23:39.685845 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:39.685806 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:39.794310 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:39.794275 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 12:23:39.794539 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:39.794331 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 12:23:40.794403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:40.794373 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 12:23:40.794837 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:40.794421 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 12:23:41.794765 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:41.794728 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 12:23:41.795115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:41.794776 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 12:23:42.794444 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:42.794401 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 12:23:42.794628 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:42.794462 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 12:23:43.794703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:43.794663 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 12:23:43.795084 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:43.794720 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 12:23:44.910744 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:44.910708 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:45.734098 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:45.734069 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:45.734554 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:45.734527 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:45.735378 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:45.735361 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:23:45.991854 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:45.991771 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:46.251714 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:46.251622 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:50.110368 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:50.110339 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:53.021666 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:53.021614 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:55.096194 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:55.096134 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:55.749257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:55.749224 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:55.995690 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:55.995654 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:57.373533 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:23:57.373462 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:23:59.372574 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:23:59.372541 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:24:00.111490 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:00.111453 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:00.867703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:00.867670 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:01.116809 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:01.116781 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:03.399818 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:03.399784 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:04.403721 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:04.403690 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:06.085428 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:06.085360 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:06.325017 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:06.324986 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:06.678979 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:06.678944 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:10.604638 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:10.604602 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:10.925999 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:10.925963 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:11.177000 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:11.176917 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:11.640924 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:11.640847 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:12.930119 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:12.930088 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:14.141041 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:14.141003 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:14.389241 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:14.389204 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:15.644756 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:15.644724 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:15.895482 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:15.895377 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:16.781551 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:16.779513 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:21.620045 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:21.620017 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:21.869040 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:21.869012 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:22.120424 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:22.120394 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:22.380209 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:22.380106 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:27.434977 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:27.434917 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:29.375761 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:24:29.375719 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:24:30.568277 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:30.568248 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:32.716725 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:32.716692 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:34.096521 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.095746 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684"] Nov 02 12:24:34.115407 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.115384 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.146122 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.146100 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-podinfo\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.146229 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.146135 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.146290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.146264 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-envoy\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.146327 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.146314 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-data\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.146395 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.146378 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-token\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.146430 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.146405 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istiod-ca-cert\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.146478 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.146438 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b6sc\" (UniqueName: \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-kube-api-access-2b6sc\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.146605 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.146583 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-certs\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.146720 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.146705 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-credential-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.177515 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.176425 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-3a78b4-openshift-default-dockercfg-84mvc\"" Nov 02 12:24:34.256535 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.243893 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684"] Nov 02 12:24:34.267087 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.250155 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-credential-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.267318 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.267298 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-podinfo\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.270950 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.270924 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.271774 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.271754 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-envoy\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.272269 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.272249 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-data\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.272796 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.272777 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-token\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.273403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.273373 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istiod-ca-cert\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.274361 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.274340 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b6sc\" (UniqueName: \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-kube-api-access-2b6sc\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.290747 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.290722 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-certs\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.291037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.274222 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istiod-ca-cert\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.291169 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.272697 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-data\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.291286 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.271611 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.291388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.267044 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-credential-socket\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.291786 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.291764 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-certs\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.336468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.336441 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-envoy\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.422710 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.403126 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b6sc\" (UniqueName: \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-kube-api-access-2b6sc\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.424145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.424116 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-token\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.424872 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.424850 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-podinfo\") pod \"e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.433694 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.433634 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:34.972305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:34.972271 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684"] Nov 02 12:24:35.008957 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:24:35.008927 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a481cdc_c7c2_417e_aebd_582ed8c6cee2.slice/crio-10545560d69273e2174b704a1ac9efb2d5746fca6199591ce2ee821bc484f8c2 WatchSource:0}: Error finding container 10545560d69273e2174b704a1ac9efb2d5746fca6199591ce2ee821bc484f8c2: Status 404 returned error can't find the container with id 10545560d69273e2174b704a1ac9efb2d5746fca6199591ce2ee821bc484f8c2 Nov 02 12:24:35.011314 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:35.011280 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:24:35.011419 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:35.011347 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:24:35.011419 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:35.011375 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:24:35.190015 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:35.189981 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" event={"ID":"2a481cdc-c7c2-417e-aebd-582ed8c6cee2","Type":"ContainerStarted","Data":"821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9"} Nov 02 12:24:35.190397 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:35.190036 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" event={"ID":"2a481cdc-c7c2-417e-aebd-582ed8c6cee2","Type":"ContainerStarted","Data":"10545560d69273e2174b704a1ac9efb2d5746fca6199591ce2ee821bc484f8c2"} Nov 02 12:24:35.311465 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:35.311365 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" podStartSLOduration=1.311351542 podStartE2EDuration="1.311351542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:24:35.309739796 +0000 UTC m=+1692.339976046" watchObservedRunningTime="2026-11-02 12:24:35.311351542 +0000 UTC m=+1692.341587792" Nov 02 12:24:35.434150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:35.434109 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:35.507559 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:35.507530 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:35.763331 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:35.763303 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:36.436035 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:36.436005 4312 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 12:24:36.436449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:36.436063 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 12:24:37.434876 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:37.434844 4312 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 12:24:37.435045 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:37.434905 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 12:24:37.905824 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:37.905739 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:38.434682 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:38.434647 4312 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 12:24:38.434891 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:38.434706 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 12:24:39.434894 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:39.434860 4312 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 12:24:39.435356 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:39.434918 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 12:24:40.434762 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:40.434723 4312 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 12:24:40.434958 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:40.434779 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 12:24:41.434687 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:41.434649 4312 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 12:24:41.434847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:41.434717 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 12:24:41.531309 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:41.531220 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:42.438710 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:42.438683 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:42.439068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:42.439049 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:42.439828 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:42.439810 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:24:43.020490 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:43.020460 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:47.516559 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.516523 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs"] Nov 02 12:24:47.519734 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.519716 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.566557 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.566530 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10768e5a-1958-4b37-b66e-8914fc33e7a9-openshift-service-ca\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.566683 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.566586 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84wjv\" (UniqueName: \"kubernetes.io/projected/10768e5a-1958-4b37-b66e-8914fc33e7a9-kube-api-access-84wjv\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.566760 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.566735 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-metrics-certs\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.566832 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.566819 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-maas-api-tls\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.571806 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.571789 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6a432197-serving-cert\"" Nov 02 12:24:47.631988 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.625081 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs"] Nov 02 12:24:47.675864 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.675826 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84wjv\" (UniqueName: \"kubernetes.io/projected/10768e5a-1958-4b37-b66e-8914fc33e7a9-kube-api-access-84wjv\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.676041 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.675918 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-metrics-certs\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.676041 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.675971 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-maas-api-tls\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.676168 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.676098 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10768e5a-1958-4b37-b66e-8914fc33e7a9-openshift-service-ca\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.676895 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.676868 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10768e5a-1958-4b37-b66e-8914fc33e7a9-openshift-service-ca\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.677637 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:24:47.677616 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-serving-cert: secret "maas-api-e2e-mt-6a432197-serving-cert" not found Nov 02 12:24:47.677828 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:24:47.677806 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-maas-api-tls podName:10768e5a-1958-4b37-b66e-8914fc33e7a9 nodeName:}" failed. No retries permitted until 2026-11-02 12:24:48.177785872 +0000 UTC m=+1705.208022114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-maas-api-tls") pod "maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" (UID: "10768e5a-1958-4b37-b66e-8914fc33e7a9") : secret "maas-api-e2e-mt-6a432197-serving-cert" not found Nov 02 12:24:47.680638 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.680615 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-metrics-certs\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.738365 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.738336 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84wjv\" (UniqueName: \"kubernetes.io/projected/10768e5a-1958-4b37-b66e-8914fc33e7a9-kube-api-access-84wjv\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:47.822664 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:47.822590 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:48.121603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:48.121514 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:48.181516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:48.181466 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-maas-api-tls\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:48.183970 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:48.183946 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-maas-api-tls\") pod \"maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:48.433467 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:48.433440 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:48.806703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:48.806674 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs"] Nov 02 12:24:48.808189 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:24:48.808158 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10768e5a_1958_4b37_b66e_8914fc33e7a9.slice/crio-408cb4fafb7825f6ec2a0f7041ef1c276d2e649a6b57abeca0fedb515693ee7a WatchSource:0}: Error finding container 408cb4fafb7825f6ec2a0f7041ef1c276d2e649a6b57abeca0fedb515693ee7a: Status 404 returned error can't find the container with id 408cb4fafb7825f6ec2a0f7041ef1c276d2e649a6b57abeca0fedb515693ee7a Nov 02 12:24:49.263153 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:49.263113 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" event={"ID":"10768e5a-1958-4b37-b66e-8914fc33e7a9","Type":"ContainerStarted","Data":"408cb4fafb7825f6ec2a0f7041ef1c276d2e649a6b57abeca0fedb515693ee7a"} Nov 02 12:24:50.273470 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:50.273411 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" event={"ID":"10768e5a-1958-4b37-b66e-8914fc33e7a9","Type":"ContainerStarted","Data":"6cbd82b479aa02000cad7ce45c65a8f13a0d42ac51c3976c8e69e2821ec1d889"} Nov 02 12:24:50.274130 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:50.274098 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:50.398673 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:50.398614 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" podStartSLOduration=2.4899468479999998 podStartE2EDuration="3.398602902s" totalImagesPullingTime="908.656054ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:24:50.398331108 +0000 UTC m=+1707.428567358" watchObservedRunningTime="2026-11-02 12:24:50.398602902 +0000 UTC m=+1707.428839151" Nov 02 12:24:53.493489 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:53.493281 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:56.285172 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:56.285140 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:24:57.765627 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:57.765590 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz"] Nov 02 12:24:57.766088 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:57.765849 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" podUID="fc5c069f-5122-4bf3-9301-0df41178c0d6" containerName="maas-api" containerID="cri-o://59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063" gracePeriod=30 Nov 02 12:24:57.969707 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:57.969655 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:58.067621 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.066602 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:24:58.174891 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.174863 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5c069f-5122-4bf3-9301-0df41178c0d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c069f-5122-4bf3-9301-0df41178c0d6-openshift-service-ca\") pod \"fc5c069f-5122-4bf3-9301-0df41178c0d6\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " Nov 02 12:24:58.175034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.174901 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5c069f-5122-4bf3-9301-0df41178c0d6-kube-api-access-rsfwk\" (UniqueName: \"kubernetes.io/projected/fc5c069f-5122-4bf3-9301-0df41178c0d6-kube-api-access-rsfwk\") pod \"fc5c069f-5122-4bf3-9301-0df41178c0d6\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " Nov 02 12:24:58.175034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.174961 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-maas-api-tls\") pod \"fc5c069f-5122-4bf3-9301-0df41178c0d6\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " Nov 02 12:24:58.175125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.175056 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-metrics-certs\") pod \"fc5c069f-5122-4bf3-9301-0df41178c0d6\" (UID: \"fc5c069f-5122-4bf3-9301-0df41178c0d6\") " Nov 02 12:24:58.175283 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.175262 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5c069f-5122-4bf3-9301-0df41178c0d6-openshift-service-ca" pod "fc5c069f-5122-4bf3-9301-0df41178c0d6" (UID: "fc5c069f-5122-4bf3-9301-0df41178c0d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:24:58.175732 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.175695 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc5c069f-5122-4bf3-9301-0df41178c0d6-openshift-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:24:58.177134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.177108 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-metrics-certs" pod "fc5c069f-5122-4bf3-9301-0df41178c0d6" (UID: "fc5c069f-5122-4bf3-9301-0df41178c0d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:24:58.177541 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.177514 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-maas-api-tls" pod "fc5c069f-5122-4bf3-9301-0df41178c0d6" (UID: "fc5c069f-5122-4bf3-9301-0df41178c0d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:24:58.177541 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.177520 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5c069f-5122-4bf3-9301-0df41178c0d6-kube-api-access-rsfwk" pod "fc5c069f-5122-4bf3-9301-0df41178c0d6" (UID: "fc5c069f-5122-4bf3-9301-0df41178c0d6"). InnerVolumeSpecName "kube-api-access-rsfwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:24:58.277234 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.277151 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsfwk\" (UniqueName: \"kubernetes.io/projected/fc5c069f-5122-4bf3-9301-0df41178c0d6-kube-api-access-rsfwk\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:24:58.277234 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.277182 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-maas-api-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:24:58.277234 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.277194 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc5c069f-5122-4bf3-9301-0df41178c0d6-metrics-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:24:58.397611 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.397576 4312 generic.go:356] "Generic (PLEG): container finished" podID="fc5c069f-5122-4bf3-9301-0df41178c0d6" containerID="59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063" exitCode=0 Nov 02 12:24:58.397807 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.397633 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" Nov 02 12:24:58.397807 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.397639 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" event={"ID":"fc5c069f-5122-4bf3-9301-0df41178c0d6","Type":"ContainerDied","Data":"59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063"} Nov 02 12:24:58.397807 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.397745 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz" event={"ID":"fc5c069f-5122-4bf3-9301-0df41178c0d6","Type":"ContainerDied","Data":"c1d1df6d7ccc89124724b7986ba992e593ffb61c7c6aaf1a57863ebe1d7b12b6"} Nov 02 12:24:58.397807 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.397767 4312 scope.go:122] "RemoveContainer" containerID="59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063" Nov 02 12:24:58.407705 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.407690 4312 scope.go:122] "RemoveContainer" containerID="59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063" Nov 02 12:24:58.407979 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:24:58.407959 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063\": container with ID starting with 59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063 not found: ID does not exist" containerID="59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063" Nov 02 12:24:58.408046 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.407988 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063"} err="failed to get container status \"59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063\": rpc error: code = NotFound desc = could not find container \"59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063\": container with ID starting with 59c747e0a9a26999563b3fc923e065b244c311a15a998c7da68987db76591063 not found: ID does not exist" Nov 02 12:24:58.562567 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.562505 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz"] Nov 02 12:24:58.569809 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:58.569787 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1cc3a7b3-5f49f9cd99-t5lxz"] Nov 02 12:24:59.074964 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:59.074930 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5c069f-5122-4bf3-9301-0df41178c0d6" path="/var/lib/kubelet/pods/fc5c069f-5122-4bf3-9301-0df41178c0d6/volumes" Nov 02 12:24:59.372965 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:24:59.372890 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:24:59.557127 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:59.557101 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:24:59.804838 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:24:59.804805 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:00.065194 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:00.065105 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:00.313029 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:00.312995 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:00.824868 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:00.824841 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:01.465860 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:01.465819 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:01.835641 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:01.835535 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:05.059635 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:05.059592 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:05.325520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:05.325416 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:05.576871 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:05.576789 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:08.144405 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:08.144368 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:08.575038 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:08.574950 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:08.829249 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:08.829151 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:10.181717 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:10.181679 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:13.306555 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:13.306480 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:13.550019 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:13.549979 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:15.287703 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:15.287656 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:20.188445 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:20.188410 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:25.085776 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:25.085745 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:29.375218 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:29.375180 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:25:29.987090 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:29.987044 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:32.900901 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:32.900865 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d"] Nov 02 12:25:32.901397 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:32.901094 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" podUID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" containerName="istio-proxy" containerID="cri-o://5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a" gracePeriod=30 Nov 02 12:25:32.950777 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:32.950748 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd"] Nov 02 12:25:32.950967 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:32.950939 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" containerID="cri-o://99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab" gracePeriod=30 Nov 02 12:25:32.955520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:32.955481 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9"] Nov 02 12:25:32.955972 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:32.955923 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" containerName="istio-proxy" containerID="cri-o://8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8" gracePeriod=30 Nov 02 12:25:33.000185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:33.000160 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf"] Nov 02 12:25:33.000365 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:33.000335 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerName="istio-proxy" containerID="cri-o://720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5" gracePeriod=30 Nov 02 12:25:34.654584 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:34.654550 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 12:25:34.655150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:34.655116 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" podUID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 12:25:35.003401 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:35.003365 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:36.059438 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:36.059363 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 12:25:36.059438 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:36.059423 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 12:25:36.720131 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:36.720098 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 12:25:36.720302 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:36.720157 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 12:25:36.765881 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:36.765854 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 12:25:36.766016 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:36.765910 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 12:25:38.024966 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:38.024938 4312 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3270a9_c28f_4705_afde_b7272562b52c.slice/crio-conmon-8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8.scope\": RecentStats: unable to find data in memory cache]" Nov 02 12:25:38.246253 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.246224 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:25:38.311757 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.311721 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-kube-api-access-wnk2h\" (UniqueName: \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-kube-api-access-wnk2h\") pod \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " Nov 02 12:25:38.311879 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.311790 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-certs\") pod \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " Nov 02 12:25:38.311879 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.311834 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-podinfo\") pod \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " Nov 02 12:25:38.311969 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.311929 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-socket\") pod \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " Nov 02 12:25:38.311969 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.311958 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-envoy\") pod \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " Nov 02 12:25:38.312044 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.312010 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-credential-socket\") pod \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " Nov 02 12:25:38.312078 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.312041 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-data\") pod \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " Nov 02 12:25:38.312111 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.312085 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-token\" (UniqueName: \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-token\") pod \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " Nov 02 12:25:38.312141 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.312114 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istiod-ca-cert\") pod \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\" (UID: \"0d5aacb6-f61a-4d84-957e-96b9241c40e2\") " Nov 02 12:25:38.312141 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.312114 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-certs" pod "0d5aacb6-f61a-4d84-957e-96b9241c40e2" (UID: "0d5aacb6-f61a-4d84-957e-96b9241c40e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.312756 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.312734 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.313452 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.313081 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-credential-socket" pod "0d5aacb6-f61a-4d84-957e-96b9241c40e2" (UID: "0d5aacb6-f61a-4d84-957e-96b9241c40e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.313452 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.313098 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-data" pod "0d5aacb6-f61a-4d84-957e-96b9241c40e2" (UID: "0d5aacb6-f61a-4d84-957e-96b9241c40e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.313452 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.313123 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istiod-ca-cert" pod "0d5aacb6-f61a-4d84-957e-96b9241c40e2" (UID: "0d5aacb6-f61a-4d84-957e-96b9241c40e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:25:38.313452 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.313435 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-socket" pod "0d5aacb6-f61a-4d84-957e-96b9241c40e2" (UID: "0d5aacb6-f61a-4d84-957e-96b9241c40e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.316647 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.316594 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-kube-api-access-wnk2h" pod "0d5aacb6-f61a-4d84-957e-96b9241c40e2" (UID: "0d5aacb6-f61a-4d84-957e-96b9241c40e2"). InnerVolumeSpecName "kube-api-access-wnk2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:25:38.316647 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.316602 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-podinfo" pod "0d5aacb6-f61a-4d84-957e-96b9241c40e2" (UID: "0d5aacb6-f61a-4d84-957e-96b9241c40e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:25:38.316647 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.316632 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-envoy" pod "0d5aacb6-f61a-4d84-957e-96b9241c40e2" (UID: "0d5aacb6-f61a-4d84-957e-96b9241c40e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.316799 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.316658 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-token" pod "0d5aacb6-f61a-4d84-957e-96b9241c40e2" (UID: "0d5aacb6-f61a-4d84-957e-96b9241c40e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:25:38.317387 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.317365 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:25:38.330791 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.330774 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:25:38.346463 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.346446 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:25:38.413642 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413621 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-podinfo\") pod \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " Nov 02 12:25:38.413748 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413651 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-data\") pod \"ab605a2c-6923-4a4c-8423-fe54a21caac0\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " Nov 02 12:25:38.413748 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413687 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-socket\") pod \"ab605a2c-6923-4a4c-8423-fe54a21caac0\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " Nov 02 12:25:38.413748 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413712 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-certs\") pod \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " Nov 02 12:25:38.413748 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413727 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-data\") pod \"ba3270a9-c28f-4705-afde-b7272562b52c\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " Nov 02 12:25:38.413748 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413746 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba3270a9-c28f-4705-afde-b7272562b52c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3270a9-c28f-4705-afde-b7272562b52c-istio-podinfo\") pod \"ba3270a9-c28f-4705-afde-b7272562b52c\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " Nov 02 12:25:38.413986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413776 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-certs\") pod \"ba3270a9-c28f-4705-afde-b7272562b52c\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " Nov 02 12:25:38.413986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413797 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-podinfo\") pod \"ab605a2c-6923-4a4c-8423-fe54a21caac0\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " Nov 02 12:25:38.413986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413841 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-token\" (UniqueName: \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-token\") pod \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " Nov 02 12:25:38.413986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413899 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-data" pod "ab605a2c-6923-4a4c-8423-fe54a21caac0" (UID: "ab605a2c-6923-4a4c-8423-fe54a21caac0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.413986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413914 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-kube-api-access-ndjz9\" (UniqueName: \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-kube-api-access-ndjz9\") pod \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " Nov 02 12:25:38.413986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.413948 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-socket\") pod \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " Nov 02 12:25:38.414289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414008 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-data\") pod \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " Nov 02 12:25:38.414289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414017 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-data" pod "ba3270a9-c28f-4705-afde-b7272562b52c" (UID: "ba3270a9-c28f-4705-afde-b7272562b52c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.414289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414042 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-socket" pod "ab605a2c-6923-4a4c-8423-fe54a21caac0" (UID: "ab605a2c-6923-4a4c-8423-fe54a21caac0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.414289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414060 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3270a9-c28f-4705-afde-b7272562b52c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3270a9-c28f-4705-afde-b7272562b52c-istiod-ca-cert\") pod \"ba3270a9-c28f-4705-afde-b7272562b52c\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " Nov 02 12:25:38.414289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414093 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-kube-api-access-5txhz\" (UniqueName: \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-kube-api-access-5txhz\") pod \"ba3270a9-c28f-4705-afde-b7272562b52c\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " Nov 02 12:25:38.414289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414095 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-certs" pod "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" (UID: "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.414289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414100 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-certs" pod "ba3270a9-c28f-4705-afde-b7272562b52c" (UID: "ba3270a9-c28f-4705-afde-b7272562b52c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.414289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414120 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-certs\") pod \"ab605a2c-6923-4a4c-8423-fe54a21caac0\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " Nov 02 12:25:38.414733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414322 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-certs" pod "ab605a2c-6923-4a4c-8423-fe54a21caac0" (UID: "ab605a2c-6923-4a4c-8423-fe54a21caac0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.414733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414486 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3270a9-c28f-4705-afde-b7272562b52c-istiod-ca-cert" pod "ba3270a9-c28f-4705-afde-b7272562b52c" (UID: "ba3270a9-c28f-4705-afde-b7272562b52c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:25:38.414733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414557 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab605a2c-6923-4a4c-8423-fe54a21caac0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab605a2c-6923-4a4c-8423-fe54a21caac0-istiod-ca-cert\") pod \"ab605a2c-6923-4a4c-8423-fe54a21caac0\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " Nov 02 12:25:38.414733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414576 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-data" pod "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" (UID: "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.414733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414619 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-credential-socket\") pod \"ba3270a9-c28f-4705-afde-b7272562b52c\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " Nov 02 12:25:38.414733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414670 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-envoy\") pod \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " Nov 02 12:25:38.414733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414703 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-istio-token\" (UniqueName: \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-istio-token\") pod \"ba3270a9-c28f-4705-afde-b7272562b52c\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " Nov 02 12:25:38.415083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414820 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-socket" pod "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" (UID: "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.415083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414842 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-envoy\") pod \"ba3270a9-c28f-4705-afde-b7272562b52c\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " Nov 02 12:25:38.415083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414883 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-token\" (UniqueName: \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-token\") pod \"ab605a2c-6923-4a4c-8423-fe54a21caac0\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " Nov 02 12:25:38.415083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414920 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-envoy\") pod \"ab605a2c-6923-4a4c-8423-fe54a21caac0\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " Nov 02 12:25:38.415083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414967 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-socket\") pod \"ba3270a9-c28f-4705-afde-b7272562b52c\" (UID: \"ba3270a9-c28f-4705-afde-b7272562b52c\") " Nov 02 12:25:38.415083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415002 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-credential-socket\") pod \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " Nov 02 12:25:38.415083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.414999 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-credential-socket" pod "ba3270a9-c28f-4705-afde-b7272562b52c" (UID: "ba3270a9-c28f-4705-afde-b7272562b52c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.415083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415033 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-credential-socket\") pod \"ab605a2c-6923-4a4c-8423-fe54a21caac0\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " Nov 02 12:25:38.415083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415067 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istiod-ca-cert\") pod \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\" (UID: \"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4\") " Nov 02 12:25:38.415486 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415109 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-kube-api-access-hj4gh\" (UniqueName: \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-kube-api-access-hj4gh\") pod \"ab605a2c-6923-4a4c-8423-fe54a21caac0\" (UID: \"ab605a2c-6923-4a4c-8423-fe54a21caac0\") " Nov 02 12:25:38.415486 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415263 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab605a2c-6923-4a4c-8423-fe54a21caac0-istiod-ca-cert" pod "ab605a2c-6923-4a4c-8423-fe54a21caac0" (UID: "ab605a2c-6923-4a4c-8423-fe54a21caac0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:25:38.415646 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415628 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415651 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415667 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415682 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415696 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415709 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415722 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415736 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415751 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415766 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415779 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415792 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415806 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3270a9-c28f-4705-afde-b7272562b52c-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415821 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415834 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab605a2c-6923-4a4c-8423-fe54a21caac0-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415848 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415862 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnk2h\" (UniqueName: \"kubernetes.io/projected/0d5aacb6-f61a-4d84-957e-96b9241c40e2-kube-api-access-wnk2h\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415878 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d5aacb6-f61a-4d84-957e-96b9241c40e2-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.415982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.415893 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.417800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.416200 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-socket" pod "ba3270a9-c28f-4705-afde-b7272562b52c" (UID: "ba3270a9-c28f-4705-afde-b7272562b52c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.417800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.416511 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-credential-socket" pod "ab605a2c-6923-4a4c-8423-fe54a21caac0" (UID: "ab605a2c-6923-4a4c-8423-fe54a21caac0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.417800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.416605 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-credential-socket" pod "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" (UID: "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.417800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.416700 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istiod-ca-cert" pod "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" (UID: "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:25:38.417800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.417078 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-podinfo" pod "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" (UID: "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:25:38.417800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.417171 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-podinfo" pod "ab605a2c-6923-4a4c-8423-fe54a21caac0" (UID: "ab605a2c-6923-4a4c-8423-fe54a21caac0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:25:38.417800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.417311 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-kube-api-access-ndjz9" pod "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" (UID: "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4"). InnerVolumeSpecName "kube-api-access-ndjz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:25:38.421479 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.421357 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-envoy" pod "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" (UID: "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.421479 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.421392 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-istio-token" pod "ba3270a9-c28f-4705-afde-b7272562b52c" (UID: "ba3270a9-c28f-4705-afde-b7272562b52c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:25:38.421479 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.421411 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-envoy" pod "ba3270a9-c28f-4705-afde-b7272562b52c" (UID: "ba3270a9-c28f-4705-afde-b7272562b52c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.421479 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.421419 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-kube-api-access-hj4gh" pod "ab605a2c-6923-4a4c-8423-fe54a21caac0" (UID: "ab605a2c-6923-4a4c-8423-fe54a21caac0"). InnerVolumeSpecName "kube-api-access-hj4gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:25:38.421479 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.421423 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-token" pod "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" (UID: "f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:25:38.421479 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.421439 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba3270a9-c28f-4705-afde-b7272562b52c-istio-podinfo" pod "ba3270a9-c28f-4705-afde-b7272562b52c" (UID: "ba3270a9-c28f-4705-afde-b7272562b52c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:25:38.421479 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.421457 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-envoy" pod "ab605a2c-6923-4a4c-8423-fe54a21caac0" (UID: "ab605a2c-6923-4a4c-8423-fe54a21caac0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:25:38.421881 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.421560 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-kube-api-access-5txhz" pod "ba3270a9-c28f-4705-afde-b7272562b52c" (UID: "ba3270a9-c28f-4705-afde-b7272562b52c"). InnerVolumeSpecName "kube-api-access-5txhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:25:38.421881 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.421744 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-token" pod "ab605a2c-6923-4a4c-8423-fe54a21caac0" (UID: "ab605a2c-6923-4a4c-8423-fe54a21caac0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:25:38.509316 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509254 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7224-6922d4"] Nov 02 12:25:38.509740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509726 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" containerName="istio-proxy" Nov 02 12:25:38.509786 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509741 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" containerName="istio-proxy" Nov 02 12:25:38.509786 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509754 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" containerName="istio-proxy" Nov 02 12:25:38.509786 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509760 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" containerName="istio-proxy" Nov 02 12:25:38.509786 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509770 4312 cpu_manager.go:414] "containerMap: removing container" podUID="fc5c069f-5122-4bf3-9301-0df41178c0d6" containerName="maas-api" Nov 02 12:25:38.509786 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509774 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5c069f-5122-4bf3-9301-0df41178c0d6" containerName="maas-api" Nov 02 12:25:38.509786 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509786 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" Nov 02 12:25:38.509998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509791 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" Nov 02 12:25:38.509998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509806 4312 cpu_manager.go:414] "containerMap: removing container" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerName="istio-proxy" Nov 02 12:25:38.509998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509811 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerName="istio-proxy" Nov 02 12:25:38.509998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509890 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerName="istio-proxy" Nov 02 12:25:38.509998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509903 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" containerName="istio-proxy" Nov 02 12:25:38.509998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509909 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" containerName="istio-proxy" Nov 02 12:25:38.509998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509916 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5c069f-5122-4bf3-9301-0df41178c0d6" containerName="maas-api" Nov 02 12:25:38.509998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.509922 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerName="istio-proxy" Nov 02 12:25:38.514388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.514373 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7224-6922d4" Nov 02 12:25:38.516996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.516974 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3270a9-c28f-4705-afde-b7272562b52c-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.516996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.516997 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517007 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517016 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjz9\" (UniqueName: \"kubernetes.io/projected/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-kube-api-access-ndjz9\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517025 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5txhz\" (UniqueName: \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-kube-api-access-5txhz\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517034 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517042 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3270a9-c28f-4705-afde-b7272562b52c-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517051 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517058 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517066 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517074 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3270a9-c28f-4705-afde-b7272562b52c-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517084 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517092 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab605a2c-6923-4a4c-8423-fe54a21caac0-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517101 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517109 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj4gh\" (UniqueName: \"kubernetes.io/projected/ab605a2c-6923-4a4c-8423-fe54a21caac0-kube-api-access-hj4gh\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.517115 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.517117 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:38.571732 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.571705 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5lqpk\"" Nov 02 12:25:38.574298 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.574273 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7224-6922d4"] Nov 02 12:25:38.609824 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.609800 4312 generic.go:356] "Generic (PLEG): container finished" podID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" containerID="720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5" exitCode=0 Nov 02 12:25:38.609936 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.609845 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" Nov 02 12:25:38.609936 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.609887 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" event={"ID":"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4","Type":"ContainerDied","Data":"720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5"} Nov 02 12:25:38.610038 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.609946 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf" event={"ID":"f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4","Type":"ContainerDied","Data":"f4482681d36440d38319030c49c63272f9de98679ef3f5c4c96ac07d92830c83"} Nov 02 12:25:38.610038 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.609980 4312 scope.go:122] "RemoveContainer" containerID="720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5" Nov 02 12:25:38.611414 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.611392 4312 generic.go:356] "Generic (PLEG): container finished" podID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" containerID="5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a" exitCode=0 Nov 02 12:25:38.611557 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.611432 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" Nov 02 12:25:38.611557 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.611486 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" event={"ID":"0d5aacb6-f61a-4d84-957e-96b9241c40e2","Type":"ContainerDied","Data":"5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a"} Nov 02 12:25:38.611667 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.611568 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d" event={"ID":"0d5aacb6-f61a-4d84-957e-96b9241c40e2","Type":"ContainerDied","Data":"989f2158f50c6150803c1bbe7bf717b0919c2229b2d2b4757926890fbf3d06fe"} Nov 02 12:25:38.619159 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.617723 4312 generic.go:356] "Generic (PLEG): container finished" podID="ba3270a9-c28f-4705-afde-b7272562b52c" containerID="8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8" exitCode=0 Nov 02 12:25:38.619159 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.617756 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" event={"ID":"ba3270a9-c28f-4705-afde-b7272562b52c","Type":"ContainerDied","Data":"8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8"} Nov 02 12:25:38.619159 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.617886 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" event={"ID":"ba3270a9-c28f-4705-afde-b7272562b52c","Type":"ContainerDied","Data":"01b2124e9784f575d0cfff1259ae3374cf98555ef72e2660deb56a890ba7973c"} Nov 02 12:25:38.619159 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.617773 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9" Nov 02 12:25:38.619159 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.617939 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2v9q\" (UniqueName: \"kubernetes.io/projected/9ecbf046-7ca3-4acb-98c7-4120e3f66a6f-kube-api-access-h2v9q\") pod \"test-curl-7224-6922d4\" (UID: \"9ecbf046-7ca3-4acb-98c7-4120e3f66a6f\") " pod="opendatahub/test-curl-7224-6922d4" Nov 02 12:25:38.619600 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.619579 4312 generic.go:356] "Generic (PLEG): container finished" podID="ab605a2c-6923-4a4c-8423-fe54a21caac0" containerID="99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab" exitCode=0 Nov 02 12:25:38.619697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.619611 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" event={"ID":"ab605a2c-6923-4a4c-8423-fe54a21caac0","Type":"ContainerDied","Data":"99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab"} Nov 02 12:25:38.619697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.619635 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" event={"ID":"ab605a2c-6923-4a4c-8423-fe54a21caac0","Type":"ContainerDied","Data":"09f0b0a8b917884aa13f1b924fff099d1b623575ba0387f38dcfda05be45da58"} Nov 02 12:25:38.619697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.619678 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd" Nov 02 12:25:38.621800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.621777 4312 scope.go:122] "RemoveContainer" containerID="720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5" Nov 02 12:25:38.622102 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:38.622082 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5\": container with ID starting with 720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5 not found: ID does not exist" containerID="720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5" Nov 02 12:25:38.622184 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.622110 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5"} err="failed to get container status \"720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5\": rpc error: code = NotFound desc = could not find container \"720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5\": container with ID starting with 720f553e0b065bd7b3bfe8e7a9a7a5f8ba7d7cddfa141f2b09b06c5f66cc9bd5 not found: ID does not exist" Nov 02 12:25:38.622184 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.622127 4312 scope.go:122] "RemoveContainer" containerID="5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a" Nov 02 12:25:38.631367 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.631353 4312 scope.go:122] "RemoveContainer" containerID="5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a" Nov 02 12:25:38.631642 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:38.631627 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a\": container with ID starting with 5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a not found: ID does not exist" containerID="5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a" Nov 02 12:25:38.631690 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.631648 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a"} err="failed to get container status \"5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a\": rpc error: code = NotFound desc = could not find container \"5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a\": container with ID starting with 5c8e5ed93a95cc37801f68b3d7be5ccd8dda4700648f5e0e936a1830dc9cda4a not found: ID does not exist" Nov 02 12:25:38.631690 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.631663 4312 scope.go:122] "RemoveContainer" containerID="8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8" Nov 02 12:25:38.639885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.639869 4312 scope.go:122] "RemoveContainer" containerID="8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8" Nov 02 12:25:38.640135 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:38.640114 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8\": container with ID starting with 8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8 not found: ID does not exist" containerID="8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8" Nov 02 12:25:38.640216 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.640137 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8"} err="failed to get container status \"8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8\": rpc error: code = NotFound desc = could not find container \"8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8\": container with ID starting with 8e4ded4c918fc07a35e68813d48e0b0d2534800baa64046a3ba296dd34e478c8 not found: ID does not exist" Nov 02 12:25:38.640216 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.640153 4312 scope.go:122] "RemoveContainer" containerID="99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab" Nov 02 12:25:38.648304 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.648283 4312 scope.go:122] "RemoveContainer" containerID="99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab" Nov 02 12:25:38.648646 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:38.648617 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab\": container with ID starting with 99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab not found: ID does not exist" containerID="99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab" Nov 02 12:25:38.648728 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.648649 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab"} err="failed to get container status \"99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab\": rpc error: code = NotFound desc = could not find container \"99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab\": container with ID starting with 99f2769e413ab08553a81b65f4fa5f30480b39fee45333f59c0e4c1fd89841ab not found: ID does not exist" Nov 02 12:25:38.718974 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.718946 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2v9q\" (UniqueName: \"kubernetes.io/projected/9ecbf046-7ca3-4acb-98c7-4120e3f66a6f-kube-api-access-h2v9q\") pod \"test-curl-7224-6922d4\" (UID: \"9ecbf046-7ca3-4acb-98c7-4120e3f66a6f\") " pod="opendatahub/test-curl-7224-6922d4" Nov 02 12:25:38.780241 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.780173 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2v9q\" (UniqueName: \"kubernetes.io/projected/9ecbf046-7ca3-4acb-98c7-4120e3f66a6f-kube-api-access-h2v9q\") pod \"test-curl-7224-6922d4\" (UID: \"9ecbf046-7ca3-4acb-98c7-4120e3f66a6f\") " pod="opendatahub/test-curl-7224-6922d4" Nov 02 12:25:38.828045 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.828012 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7224-6922d4" Nov 02 12:25:38.949823 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.949752 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d"] Nov 02 12:25:38.955708 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:38.955686 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qz97d"] Nov 02 12:25:39.065222 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:39.065158 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9"] Nov 02 12:25:39.075373 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:39.075349 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5aacb6-f61a-4d84-957e-96b9241c40e2" path="/var/lib/kubelet/pods/0d5aacb6-f61a-4d84-957e-96b9241c40e2/volumes" Nov 02 12:25:39.075656 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:39.075644 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-mvrj9"] Nov 02 12:25:39.292323 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:39.292295 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd"] Nov 02 12:25:39.298487 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:39.298459 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-bz2vd"] Nov 02 12:25:39.325231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:39.325205 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7224-6922d4"] Nov 02 12:25:39.326871 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:25:39.326848 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ecbf046_7ca3_4acb_98c7_4120e3f66a6f.slice/crio-3de4edd6762affab85868550be38ab6bff587805e204ffc7d8d7dfa978ea86cc WatchSource:0}: Error finding container 3de4edd6762affab85868550be38ab6bff587805e204ffc7d8d7dfa978ea86cc: Status 404 returned error can't find the container with id 3de4edd6762affab85868550be38ab6bff587805e204ffc7d8d7dfa978ea86cc Nov 02 12:25:39.407770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:39.407749 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf"] Nov 02 12:25:39.412017 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:39.411994 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-crsbf"] Nov 02 12:25:39.626607 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:39.626536 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7224-6922d4" event={"ID":"9ecbf046-7ca3-4acb-98c7-4120e3f66a6f","Type":"ContainerStarted","Data":"e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f"} Nov 02 12:25:39.626607 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:39.626570 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7224-6922d4" event={"ID":"9ecbf046-7ca3-4acb-98c7-4120e3f66a6f","Type":"ContainerStarted","Data":"3de4edd6762affab85868550be38ab6bff587805e204ffc7d8d7dfa978ea86cc"} Nov 02 12:25:39.740423 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:39.740382 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7224-6922d4" podStartSLOduration=1.740367045 podStartE2EDuration="1.740367045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:25:39.739566438 +0000 UTC m=+1756.769802689" watchObservedRunningTime="2026-11-02 12:25:39.740367045 +0000 UTC m=+1756.770603296" Nov 02 12:25:40.200488 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:40.200447 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:41.075939 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:41.075909 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab605a2c-6923-4a4c-8423-fe54a21caac0" path="/var/lib/kubelet/pods/ab605a2c-6923-4a4c-8423-fe54a21caac0/volumes" Nov 02 12:25:41.076357 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:41.076343 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3270a9-c28f-4705-afde-b7272562b52c" path="/var/lib/kubelet/pods/ba3270a9-c28f-4705-afde-b7272562b52c/volumes" Nov 02 12:25:41.076754 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:41.076740 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4" path="/var/lib/kubelet/pods/f4dff9c5-8b62-4a98-a162-1a6fa3c1f4b4/volumes" Nov 02 12:25:41.282453 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:41.282424 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7224-6922d4"] Nov 02 12:25:41.288072 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:41.288042 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7224-6922d4"] Nov 02 12:25:41.639438 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:41.639395 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7224-6922d4" podUID="9ecbf046-7ca3-4acb-98c7-4120e3f66a6f" containerName="test-curl-7224-6922d4" containerID="cri-o://e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f" gracePeriod=2 Nov 02 12:25:41.690506 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:41.690473 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7224-6922d4\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="9ecbf046-7ca3-4acb-98c7-4120e3f66a6f" pod="opendatahub/test-curl-7224-6922d4" Nov 02 12:25:41.944412 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:41.944386 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:43.122270 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:43.122226 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7224-6922d4\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="9ecbf046-7ca3-4acb-98c7-4120e3f66a6f" pod="opendatahub/test-curl-7224-6922d4" Nov 02 12:25:43.838119 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:43.838098 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7224-6922d4" Nov 02 12:25:43.968991 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:43.968967 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ecbf046-7ca3-4acb-98c7-4120e3f66a6f-kube-api-access-h2v9q\" (UniqueName: \"kubernetes.io/projected/9ecbf046-7ca3-4acb-98c7-4120e3f66a6f-kube-api-access-h2v9q\") pod \"9ecbf046-7ca3-4acb-98c7-4120e3f66a6f\" (UID: \"9ecbf046-7ca3-4acb-98c7-4120e3f66a6f\") " Nov 02 12:25:43.970874 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:43.970850 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ecbf046-7ca3-4acb-98c7-4120e3f66a6f-kube-api-access-h2v9q" pod "9ecbf046-7ca3-4acb-98c7-4120e3f66a6f" (UID: "9ecbf046-7ca3-4acb-98c7-4120e3f66a6f"). InnerVolumeSpecName "kube-api-access-h2v9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:25:44.069841 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.069818 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2v9q\" (UniqueName: \"kubernetes.io/projected/9ecbf046-7ca3-4acb-98c7-4120e3f66a6f-kube-api-access-h2v9q\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:25:44.656414 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.656382 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48"] Nov 02 12:25:44.656910 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.656887 4312 cpu_manager.go:414] "containerMap: removing container" podUID="9ecbf046-7ca3-4acb-98c7-4120e3f66a6f" containerName="test-curl-7224-6922d4" Nov 02 12:25:44.656910 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.656910 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ecbf046-7ca3-4acb-98c7-4120e3f66a6f" containerName="test-curl-7224-6922d4" Nov 02 12:25:44.657081 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.657067 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ecbf046-7ca3-4acb-98c7-4120e3f66a6f" containerName="test-curl-7224-6922d4" Nov 02 12:25:44.658385 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.658358 4312 generic.go:356] "Generic (PLEG): container finished" podID="9ecbf046-7ca3-4acb-98c7-4120e3f66a6f" containerID="e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f" exitCode=137 Nov 02 12:25:44.658521 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.658460 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7224-6922d4" Nov 02 12:25:44.660721 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.660703 4312 scope.go:122] "RemoveContainer" containerID="e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f" Nov 02 12:25:44.660849 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.660812 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.671323 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.671303 4312 scope.go:122] "RemoveContainer" containerID="e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f" Nov 02 12:25:44.671637 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:44.671617 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f\": container with ID starting with e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f not found: ID does not exist" containerID="e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f" Nov 02 12:25:44.671726 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.671642 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f"} err="failed to get container status \"e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f\": rpc error: code = NotFound desc = could not find container \"e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f\": container with ID starting with e903833ef1978079a15ae0b560582f81b1d7fdccda4113f87822248de532703f not found: ID does not exist" Nov 02 12:25:44.712817 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.712793 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9cbdfaf1-serving-cert\"" Nov 02 12:25:44.764635 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.764612 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48"] Nov 02 12:25:44.779727 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.779699 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-maas-api-tls\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.779841 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.779753 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmn24\" (UniqueName: \"kubernetes.io/projected/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-kube-api-access-bmn24\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.779841 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.779783 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-metrics-certs\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.779973 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.779958 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-openshift-service-ca\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.826756 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:44.826717 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7224-6922d4\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="9ecbf046-7ca3-4acb-98c7-4120e3f66a6f" pod="opendatahub/test-curl-7224-6922d4" Nov 02 12:25:44.881364 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.881342 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-openshift-service-ca\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.881491 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.881435 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-maas-api-tls\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.881491 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.881484 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmn24\" (UniqueName: \"kubernetes.io/projected/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-kube-api-access-bmn24\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.881633 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.881536 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-metrics-certs\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.881633 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:44.881580 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-serving-cert: secret "maas-api-e2e-ait-9cbdfaf1-serving-cert" not found Nov 02 12:25:44.881704 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:44.881654 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-maas-api-tls podName:2f55a865-54a3-4f5e-bbeb-e5e087c1780b nodeName:}" failed. No retries permitted until 2026-11-02 12:25:45.381634042 +0000 UTC m=+1762.411870289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-maas-api-tls") pod "maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" (UID: "2f55a865-54a3-4f5e-bbeb-e5e087c1780b") : secret "maas-api-e2e-ait-9cbdfaf1-serving-cert" not found Nov 02 12:25:44.882048 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.882022 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-openshift-service-ca\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.883904 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.883885 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-metrics-certs\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.944774 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.944750 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmn24\" (UniqueName: \"kubernetes.io/projected/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-kube-api-access-bmn24\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:44.978134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:44.978104 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:45.075634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:45.075584 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ecbf046-7ca3-4acb-98c7-4120e3f66a6f" path="/var/lib/kubelet/pods/9ecbf046-7ca3-4acb-98c7-4120e3f66a6f/volumes" Nov 02 12:25:45.221774 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:45.221692 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:45.386998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:45.386971 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-maas-api-tls\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:45.389385 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:45.389362 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-maas-api-tls\") pod \"maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:45.468689 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:45.468662 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:45.579052 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:45.579002 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:45.953121 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:45.953095 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48"] Nov 02 12:25:45.955436 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:25:45.955407 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f55a865_54a3_4f5e_bbeb_e5e087c1780b.slice/crio-62b7a725766fb917635f1393ec47fc0a5585b163cfd92c82f0f35a22265edbfe WatchSource:0}: Error finding container 62b7a725766fb917635f1393ec47fc0a5585b163cfd92c82f0f35a22265edbfe: Status 404 returned error can't find the container with id 62b7a725766fb917635f1393ec47fc0a5585b163cfd92c82f0f35a22265edbfe Nov 02 12:25:46.675642 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:46.675610 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" event={"ID":"2f55a865-54a3-4f5e-bbeb-e5e087c1780b","Type":"ContainerStarted","Data":"62b7a725766fb917635f1393ec47fc0a5585b163cfd92c82f0f35a22265edbfe"} Nov 02 12:25:46.806381 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:46.804208 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm"] Nov 02 12:25:46.978345 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:46.978264 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:47.683137 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:47.683095 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" event={"ID":"2f55a865-54a3-4f5e-bbeb-e5e087c1780b","Type":"ContainerStarted","Data":"1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4"} Nov 02 12:25:47.683311 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:47.683164 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:47.802476 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:47.802421 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" podStartSLOduration=2.987050743 podStartE2EDuration="3.802407315s" totalImagesPullingTime="815.356572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:25:47.800040505 +0000 UTC m=+1764.830276757" watchObservedRunningTime="2026-11-02 12:25:47.802407315 +0000 UTC m=+1764.832643564" Nov 02 12:25:50.082912 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:50.082884 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:53.695520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:53.695470 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:25:54.992040 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:54.992001 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:25:59.372372 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:25:59.372332 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:25:59.984765 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:25:59.984735 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:01.852150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:01.852104 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" podUID="b31d6ac5-c822-4337-ab22-a768627bb288" containerName="main" containerID="cri-o://c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf" gracePeriod=45 Nov 02 12:26:02.110768 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.110701 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:26:02.152282 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.152251 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-dshm\") pod \"b31d6ac5-c822-4337-ab22-a768627bb288\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " Nov 02 12:26:02.152457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.152314 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-tmp-dir\") pod \"b31d6ac5-c822-4337-ab22-a768627bb288\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " Nov 02 12:26:02.152457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.152380 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31d6ac5-c822-4337-ab22-a768627bb288-kube-api-access-6mjgh\" (UniqueName: \"kubernetes.io/projected/b31d6ac5-c822-4337-ab22-a768627bb288-kube-api-access-6mjgh\") pod \"b31d6ac5-c822-4337-ab22-a768627bb288\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " Nov 02 12:26:02.152457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.152441 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-model-cache\") pod \"b31d6ac5-c822-4337-ab22-a768627bb288\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " Nov 02 12:26:02.152674 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.152546 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-home\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-home\") pod \"b31d6ac5-c822-4337-ab22-a768627bb288\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " Nov 02 12:26:02.152674 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.152569 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b31d6ac5-c822-4337-ab22-a768627bb288-tls-certs\" (UniqueName: \"kubernetes.io/secret/b31d6ac5-c822-4337-ab22-a768627bb288-tls-certs\") pod \"b31d6ac5-c822-4337-ab22-a768627bb288\" (UID: \"b31d6ac5-c822-4337-ab22-a768627bb288\") " Nov 02 12:26:02.152674 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.152658 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-tmp-dir" pod "b31d6ac5-c822-4337-ab22-a768627bb288" (UID: "b31d6ac5-c822-4337-ab22-a768627bb288"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:26:02.152836 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.152757 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-model-cache" pod "b31d6ac5-c822-4337-ab22-a768627bb288" (UID: "b31d6ac5-c822-4337-ab22-a768627bb288"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:26:02.152886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.152855 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-home" pod "b31d6ac5-c822-4337-ab22-a768627bb288" (UID: "b31d6ac5-c822-4337-ab22-a768627bb288"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:26:02.153108 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.153039 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:26:02.153108 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.153064 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:26:02.153108 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.153079 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:26:02.154702 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.154680 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-dshm" pod "b31d6ac5-c822-4337-ab22-a768627bb288" (UID: "b31d6ac5-c822-4337-ab22-a768627bb288"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:26:02.154815 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.154801 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31d6ac5-c822-4337-ab22-a768627bb288-tls-certs" pod "b31d6ac5-c822-4337-ab22-a768627bb288" (UID: "b31d6ac5-c822-4337-ab22-a768627bb288"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:26:02.154864 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.154822 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31d6ac5-c822-4337-ab22-a768627bb288-kube-api-access-6mjgh" pod "b31d6ac5-c822-4337-ab22-a768627bb288" (UID: "b31d6ac5-c822-4337-ab22-a768627bb288"). InnerVolumeSpecName "kube-api-access-6mjgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:26:02.253996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.253968 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b31d6ac5-c822-4337-ab22-a768627bb288-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:26:02.253996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.253994 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mjgh\" (UniqueName: \"kubernetes.io/projected/b31d6ac5-c822-4337-ab22-a768627bb288-kube-api-access-6mjgh\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:26:02.253996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.254004 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b31d6ac5-c822-4337-ab22-a768627bb288-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:26:02.760236 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.760204 4312 generic.go:356] "Generic (PLEG): container finished" podID="b31d6ac5-c822-4337-ab22-a768627bb288" containerID="c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf" exitCode=0 Nov 02 12:26:02.760403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.760246 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" Nov 02 12:26:02.760403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.760279 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" event={"ID":"b31d6ac5-c822-4337-ab22-a768627bb288","Type":"ContainerDied","Data":"c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf"} Nov 02 12:26:02.760403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.760316 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm" event={"ID":"b31d6ac5-c822-4337-ab22-a768627bb288","Type":"ContainerDied","Data":"ef28e2d13de89857b3c35f712b48303caadd161c70dc00c7109ab897159a3332"} Nov 02 12:26:02.760403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.760333 4312 scope.go:122] "RemoveContainer" containerID="c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf" Nov 02 12:26:02.770565 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.770549 4312 scope.go:122] "RemoveContainer" containerID="c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf" Nov 02 12:26:02.770819 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:26:02.770800 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf\": container with ID starting with c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf not found: ID does not exist" containerID="c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf" Nov 02 12:26:02.770883 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.770826 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf"} err="failed to get container status \"c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf\": rpc error: code = NotFound desc = could not find container \"c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf\": container with ID starting with c53e906dd7ae2c54a9e80f02a9bc8bbb56392a32bc94da0d5ed038458d2f92bf not found: ID does not exist" Nov 02 12:26:02.924792 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.924768 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm"] Nov 02 12:26:02.974308 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:02.974285 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6a432197/e2e-lifecycle-model-6a432197-kserve-bc76d97df-cpdfm"] Nov 02 12:26:03.075250 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:03.075191 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31d6ac5-c822-4337-ab22-a768627bb288" path="/var/lib/kubelet/pods/b31d6ac5-c822-4337-ab22-a768627bb288/volumes" Nov 02 12:26:05.062435 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:05.062401 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:09.909069 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:09.909038 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:13.622519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:13.622473 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:14.818949 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:14.818913 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:19.700019 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:19.699986 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:22.615414 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:22.615367 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:22.948313 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:22.948286 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:23.612693 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:23.612612 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:23.863093 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:23.863006 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:24.451824 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:24.451793 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:24.698538 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:24.698481 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:24.943002 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:24.942966 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:29.372764 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:26:29.372727 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:26:30.215154 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:30.215121 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:30.473510 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:30.473404 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:35.350725 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:35.350692 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:40.678476 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:40.678425 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:45.827249 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:45.827210 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:50.423906 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:50.423868 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:50.675605 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:50.675517 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:50.922970 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:50.922932 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:55.793384 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:55.793345 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:26:59.372394 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:26:59.372359 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:26:59.819789 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:59.819760 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs"] Nov 02 12:26:59.819978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:26:59.819958 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" podUID="10768e5a-1958-4b37-b66e-8914fc33e7a9" containerName="maas-api" containerID="cri-o://6cbd82b479aa02000cad7ce45c65a8f13a0d42ac51c3976c8e69e2821ec1d889" gracePeriod=30 Nov 02 12:27:00.026305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.026073 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:00.057456 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.057417 4312 generic.go:356] "Generic (PLEG): container finished" podID="10768e5a-1958-4b37-b66e-8914fc33e7a9" containerID="6cbd82b479aa02000cad7ce45c65a8f13a0d42ac51c3976c8e69e2821ec1d889" exitCode=0 Nov 02 12:27:00.057620 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.057521 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" event={"ID":"10768e5a-1958-4b37-b66e-8914fc33e7a9","Type":"ContainerDied","Data":"6cbd82b479aa02000cad7ce45c65a8f13a0d42ac51c3976c8e69e2821ec1d889"} Nov 02 12:27:00.083559 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.083477 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:27:00.201612 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.201581 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10768e5a-1958-4b37-b66e-8914fc33e7a9-kube-api-access-84wjv\" (UniqueName: \"kubernetes.io/projected/10768e5a-1958-4b37-b66e-8914fc33e7a9-kube-api-access-84wjv\") pod \"10768e5a-1958-4b37-b66e-8914fc33e7a9\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " Nov 02 12:27:00.201802 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.201625 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10768e5a-1958-4b37-b66e-8914fc33e7a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10768e5a-1958-4b37-b66e-8914fc33e7a9-openshift-service-ca\") pod \"10768e5a-1958-4b37-b66e-8914fc33e7a9\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " Nov 02 12:27:00.201802 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.201643 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-metrics-certs\") pod \"10768e5a-1958-4b37-b66e-8914fc33e7a9\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " Nov 02 12:27:00.201802 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.201749 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-maas-api-tls\") pod \"10768e5a-1958-4b37-b66e-8914fc33e7a9\" (UID: \"10768e5a-1958-4b37-b66e-8914fc33e7a9\") " Nov 02 12:27:00.202014 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.201988 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10768e5a-1958-4b37-b66e-8914fc33e7a9-openshift-service-ca" pod "10768e5a-1958-4b37-b66e-8914fc33e7a9" (UID: "10768e5a-1958-4b37-b66e-8914fc33e7a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:27:00.203512 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.203299 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10768e5a-1958-4b37-b66e-8914fc33e7a9-openshift-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:27:00.204134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.204105 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-metrics-certs" pod "10768e5a-1958-4b37-b66e-8914fc33e7a9" (UID: "10768e5a-1958-4b37-b66e-8914fc33e7a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:27:00.204134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.204111 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10768e5a-1958-4b37-b66e-8914fc33e7a9-kube-api-access-84wjv" pod "10768e5a-1958-4b37-b66e-8914fc33e7a9" (UID: "10768e5a-1958-4b37-b66e-8914fc33e7a9"). InnerVolumeSpecName "kube-api-access-84wjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:27:00.204264 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.204223 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-maas-api-tls" pod "10768e5a-1958-4b37-b66e-8914fc33e7a9" (UID: "10768e5a-1958-4b37-b66e-8914fc33e7a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:27:00.304928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.304892 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84wjv\" (UniqueName: \"kubernetes.io/projected/10768e5a-1958-4b37-b66e-8914fc33e7a9-kube-api-access-84wjv\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:27:00.304928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.304924 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-metrics-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:27:00.304928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:00.304934 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10768e5a-1958-4b37-b66e-8914fc33e7a9-maas-api-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:27:01.065083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:01.065057 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" Nov 02 12:27:01.065740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:01.065053 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs" event={"ID":"10768e5a-1958-4b37-b66e-8914fc33e7a9","Type":"ContainerDied","Data":"408cb4fafb7825f6ec2a0f7041ef1c276d2e649a6b57abeca0fedb515693ee7a"} Nov 02 12:27:01.065740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:01.065208 4312 scope.go:122] "RemoveContainer" containerID="6cbd82b479aa02000cad7ce45c65a8f13a0d42ac51c3976c8e69e2821ec1d889" Nov 02 12:27:01.228192 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:01.228166 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs"] Nov 02 12:27:01.243950 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:01.243929 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6a432197-79f4f6646c-t6bqs"] Nov 02 12:27:01.883836 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:01.883808 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:02.265959 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:02.265928 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:02.507585 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:02.507548 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:03.075975 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:03.075940 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10768e5a-1958-4b37-b66e-8914fc33e7a9" path="/var/lib/kubelet/pods/10768e5a-1958-4b37-b66e-8914fc33e7a9/volumes" Nov 02 12:27:03.855928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:03.855897 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:04.490627 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:04.490597 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:07.120713 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:07.120645 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:11.164634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.164597 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48"] Nov 02 12:27:11.165079 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.164832 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" podUID="2f55a865-54a3-4f5e-bbeb-e5e087c1780b" containerName="maas-api" containerID="cri-o://1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4" gracePeriod=30 Nov 02 12:27:11.380087 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.380056 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:11.455511 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.455471 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:27:11.612313 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.612048 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-openshift-service-ca\") pod \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " Nov 02 12:27:11.612313 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.612113 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-kube-api-access-bmn24\" (UniqueName: \"kubernetes.io/projected/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-kube-api-access-bmn24\") pod \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " Nov 02 12:27:11.612590 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.612285 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-metrics-certs\") pod \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " Nov 02 12:27:11.612590 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.612459 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-maas-api-tls\") pod \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\" (UID: \"2f55a865-54a3-4f5e-bbeb-e5e087c1780b\") " Nov 02 12:27:11.613847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.613815 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-openshift-service-ca" pod "2f55a865-54a3-4f5e-bbeb-e5e087c1780b" (UID: "2f55a865-54a3-4f5e-bbeb-e5e087c1780b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:27:11.614373 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.614347 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-openshift-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:27:11.615701 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.615667 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-maas-api-tls" pod "2f55a865-54a3-4f5e-bbeb-e5e087c1780b" (UID: "2f55a865-54a3-4f5e-bbeb-e5e087c1780b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:27:11.615988 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.615962 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-kube-api-access-bmn24" pod "2f55a865-54a3-4f5e-bbeb-e5e087c1780b" (UID: "2f55a865-54a3-4f5e-bbeb-e5e087c1780b"). InnerVolumeSpecName "kube-api-access-bmn24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:27:11.616708 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.616659 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-metrics-certs" pod "2f55a865-54a3-4f5e-bbeb-e5e087c1780b" (UID: "2f55a865-54a3-4f5e-bbeb-e5e087c1780b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:27:11.716247 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.716165 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmn24\" (UniqueName: \"kubernetes.io/projected/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-kube-api-access-bmn24\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:27:11.716247 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.716197 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-metrics-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:27:11.716247 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:11.716207 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f55a865-54a3-4f5e-bbeb-e5e087c1780b-maas-api-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:27:12.140710 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:12.140603 4312 generic.go:356] "Generic (PLEG): container finished" podID="2f55a865-54a3-4f5e-bbeb-e5e087c1780b" containerID="1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4" exitCode=0 Nov 02 12:27:12.140710 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:12.140668 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" event={"ID":"2f55a865-54a3-4f5e-bbeb-e5e087c1780b","Type":"ContainerDied","Data":"1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4"} Nov 02 12:27:12.140946 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:12.140719 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" event={"ID":"2f55a865-54a3-4f5e-bbeb-e5e087c1780b","Type":"ContainerDied","Data":"62b7a725766fb917635f1393ec47fc0a5585b163cfd92c82f0f35a22265edbfe"} Nov 02 12:27:12.140946 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:12.140746 4312 scope.go:122] "RemoveContainer" containerID="1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4" Nov 02 12:27:12.141271 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:12.141249 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48" Nov 02 12:27:12.156584 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:12.156550 4312 scope.go:122] "RemoveContainer" containerID="1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4" Nov 02 12:27:12.157015 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:27:12.156987 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4\": container with ID starting with 1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4 not found: ID does not exist" containerID="1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4" Nov 02 12:27:12.157137 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:12.157067 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4"} err="failed to get container status \"1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4\": rpc error: code = NotFound desc = could not find container \"1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4\": container with ID starting with 1a3ca936650329e86edfd27c08c3f1bb7122527edbad5e1f70e847ca1167a3d4 not found: ID does not exist" Nov 02 12:27:12.305396 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:12.305367 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48"] Nov 02 12:27:12.355093 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:12.355069 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9cbdfaf1-57bdfff9c9-f9t48"] Nov 02 12:27:12.960948 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:12.960917 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:13.075946 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:13.075914 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f55a865-54a3-4f5e-bbeb-e5e087c1780b" path="/var/lib/kubelet/pods/2f55a865-54a3-4f5e-bbeb-e5e087c1780b/volumes" Nov 02 12:27:13.206828 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:13.206799 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:13.478293 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:13.478196 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:14.008512 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:14.008459 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:14.944866 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:14.944833 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:15.585523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:15.581992 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:16.018662 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.018626 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp"] Nov 02 12:27:16.019283 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.019258 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2f55a865-54a3-4f5e-bbeb-e5e087c1780b" containerName="maas-api" Nov 02 12:27:16.019375 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.019287 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f55a865-54a3-4f5e-bbeb-e5e087c1780b" containerName="maas-api" Nov 02 12:27:16.019375 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.019310 4312 cpu_manager.go:414] "containerMap: removing container" podUID="b31d6ac5-c822-4337-ab22-a768627bb288" containerName="main" Nov 02 12:27:16.019375 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.019319 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31d6ac5-c822-4337-ab22-a768627bb288" containerName="main" Nov 02 12:27:16.019375 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.019361 4312 cpu_manager.go:414] "containerMap: removing container" podUID="10768e5a-1958-4b37-b66e-8914fc33e7a9" containerName="maas-api" Nov 02 12:27:16.019375 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.019369 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10768e5a-1958-4b37-b66e-8914fc33e7a9" containerName="maas-api" Nov 02 12:27:16.019700 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.019481 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="10768e5a-1958-4b37-b66e-8914fc33e7a9" containerName="maas-api" Nov 02 12:27:16.019700 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.019515 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f55a865-54a3-4f5e-bbeb-e5e087c1780b" containerName="maas-api" Nov 02 12:27:16.019700 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.019526 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31d6ac5-c822-4337-ab22-a768627bb288" containerName="main" Nov 02 12:27:16.025271 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.025249 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.246524 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.207650 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ccee868d-openshift-default-dockercfg-hzvpl\"" Nov 02 12:27:16.349578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.347108 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-certs\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.349578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.347169 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-podinfo\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.349578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.347211 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-credential-socket\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.349578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.347252 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-envoy\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.349578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.347318 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7w5k\" (UniqueName: \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-kube-api-access-f7w5k\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.349578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.347354 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-data\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.349578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.347382 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-token\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.349578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.347441 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad378898-e30c-4945-9b5f-a310b62ec3b8-istiod-ca-cert\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.349578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.347550 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-socket\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.374435 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.374404 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp"] Nov 02 12:27:16.385761 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.385733 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:16.452466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.452423 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-token\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.452738 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.452716 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad378898-e30c-4945-9b5f-a310b62ec3b8-istiod-ca-cert\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.452975 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.452957 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-socket\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.453149 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.453124 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-certs\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.453296 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.453277 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-podinfo\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.453466 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.453434 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-credential-socket\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.453634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.453617 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-envoy\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.453825 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.453809 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7w5k\" (UniqueName: \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-kube-api-access-f7w5k\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.453983 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.453965 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-data\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.454638 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.454618 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-data\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.455857 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.455804 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-socket\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.455969 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.455864 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-certs\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.456610 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.456574 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-credential-socket\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.457817 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.457795 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad378898-e30c-4945-9b5f-a310b62ec3b8-istiod-ca-cert\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.469781 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.469726 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-envoy\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.502527 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.500322 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-podinfo\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.522267 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.522205 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-token\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.525294 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.525275 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7w5k\" (UniqueName: \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-kube-api-access-f7w5k\") pod \"e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.656399 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.655688 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:16.781027 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.780632 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxd45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 12:27:16.781027 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.780711 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 12:27:16.841872 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.841666 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lvhwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 12:27:16.841872 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:16.841739 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 12:27:17.141193 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:17.141058 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:17.376642 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:17.376616 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp"] Nov 02 12:27:17.381531 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:17.380673 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:27:17.381531 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:17.380754 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:27:17.381531 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:17.380799 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:27:17.417231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:17.417151 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" event={"ID":"ad378898-e30c-4945-9b5f-a310b62ec3b8","Type":"ContainerStarted","Data":"d65214561fc573829456c3f79040a1e12c71acfed140006f81460ce7a3bd2e58"} Nov 02 12:27:18.424471 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:18.424414 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" event={"ID":"ad378898-e30c-4945-9b5f-a310b62ec3b8","Type":"ContainerStarted","Data":"d17b8c45bafea9589afc776a730c94d01ab05f8054ff4adda2f9afb88e81de45"} Nov 02 12:27:18.630233 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:18.630184 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" podStartSLOduration=3.630171072 podStartE2EDuration="3.630171072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:18.595116562 +0000 UTC m=+1855.625352811" watchObservedRunningTime="2026-11-02 12:27:18.630171072 +0000 UTC m=+1855.660407325" Nov 02 12:27:18.656468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:18.656445 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:18.661604 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:18.661580 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:19.430639 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:19.430607 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:19.431767 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:19.431744 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:27:19.479470 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:19.479438 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:20.929949 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:20.929910 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:21.187194 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:21.187120 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:22.227221 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.227182 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj"] Nov 02 12:27:22.230874 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.230852 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.247179 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.247160 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:22.282534 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.282487 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.282619 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.282587 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.282619 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.282615 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.282705 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.282637 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkzc\" (UniqueName: \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-kube-api-access-8pkzc\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.282705 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.282663 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.282818 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.282797 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.282885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.282869 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.282920 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.282909 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.282997 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.282976 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.335768 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.335746 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj"] Nov 02 12:27:22.384346 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384321 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.384477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384365 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkzc\" (UniqueName: \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-kube-api-access-8pkzc\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.384477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384402 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.384617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384480 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.384617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384557 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.384617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384594 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.384779 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384632 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.384779 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384663 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.384779 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384722 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.384916 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384872 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-data\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.384973 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384924 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-certs\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.385029 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.384992 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-credential-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.385134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.385116 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-socket\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.385334 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.385315 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istiod-ca-cert\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.386830 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.386809 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-envoy\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.386920 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.386882 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-podinfo\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.447431 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.447409 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-token\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.447652 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.447634 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkzc\" (UniqueName: \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-kube-api-access-8pkzc\") pod \"e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.497649 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.497598 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:22.544590 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.544566 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:22.975085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.975057 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj"] Nov 02 12:27:22.976668 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:27:22.976646 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72aa094f_f655_4b75_8a90_f1a2536ce9d6.slice/crio-bac0e7007070503dccca5d31ceb0af1e5a9b36e0e1265be417b14ed8cc3c1105 WatchSource:0}: Error finding container bac0e7007070503dccca5d31ceb0af1e5a9b36e0e1265be417b14ed8cc3c1105: Status 404 returned error can't find the container with id bac0e7007070503dccca5d31ceb0af1e5a9b36e0e1265be417b14ed8cc3c1105 Nov 02 12:27:22.978949 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.978919 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:27:22.979042 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.978980 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:27:22.979042 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:22.979007 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:27:23.478074 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:23.478034 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" event={"ID":"72aa094f-f655-4b75-8a90-f1a2536ce9d6","Type":"ContainerStarted","Data":"fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c"} Nov 02 12:27:23.478565 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:23.478097 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" event={"ID":"72aa094f-f655-4b75-8a90-f1a2536ce9d6","Type":"ContainerStarted","Data":"bac0e7007070503dccca5d31ceb0af1e5a9b36e0e1265be417b14ed8cc3c1105"} Nov 02 12:27:23.521758 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:23.521728 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:23.544926 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:23.544902 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:23.774055 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:23.774008 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:24.137699 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.137580 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:24.394707 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.394592 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:24.546983 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.546949 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 12:27:24.547516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.547008 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" podUID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 12:27:24.585527 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.585158 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" podStartSLOduration=2.5851350010000003 podStartE2EDuration="2.585135001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:23.658850641 +0000 UTC m=+1860.689086891" watchObservedRunningTime="2026-11-02 12:27:24.585135001 +0000 UTC m=+1861.615371252" Nov 02 12:27:24.592798 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.592765 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g"] Nov 02 12:27:24.598333 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.598309 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:27:24.646714 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.646632 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g"] Nov 02 12:27:24.665519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.660605 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ccee868d\"" Nov 02 12:27:24.699936 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.699890 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ccee868d-dockercfg-4vft4\"" Nov 02 12:27:24.708652 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.708623 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g\" (UID: \"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8\") " pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:27:24.708857 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.708838 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl2x7\" (UniqueName: \"kubernetes.io/projected/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-kube-api-access-cl2x7\") pod \"payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g\" (UID: \"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8\") " pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:27:24.810222 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.810191 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2x7\" (UniqueName: \"kubernetes.io/projected/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-kube-api-access-cl2x7\") pod \"payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g\" (UID: \"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8\") " pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:27:24.810414 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.810245 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g\" (UID: \"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8\") " pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:27:24.811020 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.811001 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-plugins-config-volume\") pod \"payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g\" (UID: \"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8\") " pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:27:24.872521 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.871931 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2x7\" (UniqueName: \"kubernetes.io/projected/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-kube-api-access-cl2x7\") pod \"payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g\" (UID: \"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8\") " pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:27:24.913094 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:24.913071 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:27:25.296443 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:25.296346 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:25.303663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:25.303642 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g"] Nov 02 12:27:25.306027 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:27:25.306000 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf4e46c_ad51_4527_8cfb_3c01f31da9d8.slice/crio-9a7c94baff4ac3eee8dc59ee11266122ea36eba69ada6ea77c424689b8757402 WatchSource:0}: Error finding container 9a7c94baff4ac3eee8dc59ee11266122ea36eba69ada6ea77c424689b8757402: Status 404 returned error can't find the container with id 9a7c94baff4ac3eee8dc59ee11266122ea36eba69ada6ea77c424689b8757402 Nov 02 12:27:25.495999 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:25.495958 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" event={"ID":"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8","Type":"ContainerStarted","Data":"9a7c94baff4ac3eee8dc59ee11266122ea36eba69ada6ea77c424689b8757402"} Nov 02 12:27:25.545847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:25.545821 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 12:27:25.545996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:25.545873 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" podUID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 12:27:26.545864 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:26.545832 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 12:27:26.546335 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:26.545888 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" podUID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 12:27:27.545889 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:27.545855 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 12:27:27.546322 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:27.545905 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" podUID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 12:27:27.774648 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:27.774579 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:28.025123 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:28.025088 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:28.545649 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:28.545622 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Nov 02 12:27:28.545755 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:28.545669 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" podUID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Nov 02 12:27:29.373234 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:27:29.373191 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:27:29.544258 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:29.543232 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" event={"ID":"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8","Type":"ContainerStarted","Data":"81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8"} Nov 02 12:27:29.544258 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:29.544199 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:27:29.564069 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:29.563991 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:29.666872 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:29.666812 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" podStartSLOduration=2.54448806 podStartE2EDuration="5.666793143s" totalImagesPullingTime="3.122305083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:29.662334905 +0000 UTC m=+1866.692571155" watchObservedRunningTime="2026-11-02 12:27:29.666793143 +0000 UTC m=+1866.697029395" Nov 02 12:27:30.284019 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:30.283935 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:30.409525 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:30.409476 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:30.410041 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:30.410022 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:30.411118 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:30.411096 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:27:34.587546 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.587513 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr"] Nov 02 12:27:34.591324 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.591307 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.643650 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.643625 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"openshift-service-ca.crt\"" Nov 02 12:27:34.693042 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.693019 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-3a78b4\"" Nov 02 12:27:34.693356 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.693334 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-50db61f22b2e\"" Nov 02 12:27:34.693427 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.693332 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-14e0f93131b7\"" Nov 02 12:27:34.693427 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.693372 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-4b8a830d91a3\"" Nov 02 12:27:34.693427 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.693390 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"routing-overlay\"" Nov 02 12:27:34.693427 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.693402 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-134e84d82424\"" Nov 02 12:27:34.693636 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.693458 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"payload-processing-external-model-e2e--a8ab13a3-dockercfg-wmcbx\"" Nov 02 12:27:34.693636 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.693512 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-dcd507907f10\"" Nov 02 12:27:34.695724 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.695706 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr"] Nov 02 12:27:34.743086 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.743055 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.743218 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.743166 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.743283 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.743231 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.743283 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.743251 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rbg6\" (UniqueName: \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-kube-api-access-6rbg6\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.743795 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.743779 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"kube-root-ca.crt\"" Nov 02 12:27:34.844571 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.844484 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.844571 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.844540 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rbg6\" (UniqueName: \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-kube-api-access-6rbg6\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.844724 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.844603 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.844724 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.844653 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.845370 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.845344 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.845464 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.845380 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.847331 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.847311 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:34.905095 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:34.905068 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rbg6\" (UniqueName: \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-kube-api-access-6rbg6\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:35.203912 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:35.203887 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:35.273398 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:35.273296 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:35.585447 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:35.585409 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr"] Nov 02 12:27:35.588810 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:27:35.588783 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b75ed7_022a_4d07_9d7b_a699d52d087a.slice/crio-2d8192ade6fefb89a4337bdd31b558df8ba4d0e0cdb19b18f26203a6e8096f17 WatchSource:0}: Error finding container 2d8192ade6fefb89a4337bdd31b558df8ba4d0e0cdb19b18f26203a6e8096f17: Status 404 returned error can't find the container with id 2d8192ade6fefb89a4337bdd31b558df8ba4d0e0cdb19b18f26203a6e8096f17 Nov 02 12:27:36.585813 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:36.585781 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" event={"ID":"91b75ed7-022a-4d07-9d7b-a699d52d087a","Type":"ContainerStarted","Data":"dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931"} Nov 02 12:27:36.586016 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:36.585833 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" event={"ID":"91b75ed7-022a-4d07-9d7b-a699d52d087a","Type":"ContainerStarted","Data":"2d8192ade6fefb89a4337bdd31b558df8ba4d0e0cdb19b18f26203a6e8096f17"} Nov 02 12:27:36.586016 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:36.585871 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:36.704228 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:36.704185 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" podStartSLOduration=2.704170987 podStartE2EDuration="2.704170987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:36.702116961 +0000 UTC m=+1873.732353215" watchObservedRunningTime="2026-11-02 12:27:36.704170987 +0000 UTC m=+1873.734407237" Nov 02 12:27:40.188677 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:40.188642 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:41.221361 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:41.221332 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:41.562876 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:41.562806 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:27:45.535187 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:45.535150 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:47.211482 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:47.211449 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:47.594889 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:47.594801 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:27:50.547648 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:50.547613 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:55.302886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.302842 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm"] Nov 02 12:27:55.316519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.316281 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.370820 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.370786 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm"] Nov 02 12:27:55.392520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.391180 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.392520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.391347 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-metrics-certs\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.392520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.391381 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5v9w\" (UniqueName: \"kubernetes.io/projected/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-kube-api-access-r5v9w\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.392520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.391481 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-openshift-service-ca\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.434350 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.432719 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ccee868d-serving-cert\"" Nov 02 12:27:55.533371 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.533317 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-metrics-certs\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.533371 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.533373 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5v9w\" (UniqueName: \"kubernetes.io/projected/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-kube-api-access-r5v9w\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.533658 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.533472 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-openshift-service-ca\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.533658 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.533548 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.533882 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:27:55.533859 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-serving-cert: secret "maas-api-e2e-ait-ccee868d-serving-cert" not found Nov 02 12:27:55.534079 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:27:55.534064 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls podName:01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb nodeName:}" failed. No retries permitted until 2026-11-02 12:27:56.034040248 +0000 UTC m=+1893.064276495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls") pod "maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" (UID: "01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb") : secret "maas-api-e2e-ait-ccee868d-serving-cert" not found Nov 02 12:27:55.535573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.535301 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-openshift-service-ca\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.543236 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.543214 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-metrics-certs\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.605408 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.605334 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5v9w\" (UniqueName: \"kubernetes.io/projected/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-kube-api-access-r5v9w\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:55.607963 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.607932 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:55.872569 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:55.871853 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:56.040831 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:56.040798 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:56.041002 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:27:56.040953 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-serving-cert: secret "maas-api-e2e-ait-ccee868d-serving-cert" not found Nov 02 12:27:56.041043 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:27:56.041036 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls podName:01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb nodeName:}" failed. No retries permitted until 2026-11-02 12:27:57.041020906 +0000 UTC m=+1894.071257149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls") pod "maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" (UID: "01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb") : secret "maas-api-e2e-ait-ccee868d-serving-cert" not found Nov 02 12:27:56.301027 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:56.300974 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:27:57.071520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:57.071251 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:57.121721 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:57.121699 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls\") pod \"maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:57.162776 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:57.159644 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:57.759667 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:57.759631 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm"] Nov 02 12:27:58.783459 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:58.783418 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" event={"ID":"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb","Type":"ContainerStarted","Data":"00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778"} Nov 02 12:27:58.783871 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:58.783485 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" event={"ID":"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb","Type":"ContainerStarted","Data":"823c8d0a6bc3b3c9537ee4db085caec88e6ae71d1e1416c0e44428b5d52b214d"} Nov 02 12:27:58.783871 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:58.783554 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:27:58.901062 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:27:58.900999 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" podStartSLOduration=3.001222056 podStartE2EDuration="3.900982756s" totalImagesPullingTime="899.7607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:58.900722584 +0000 UTC m=+1895.930958834" watchObservedRunningTime="2026-11-02 12:27:58.900982756 +0000 UTC m=+1895.931219006" Nov 02 12:27:59.371972 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:27:59.371936 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:28:01.086605 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:01.086574 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:04.801803 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:04.801776 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:28:06.073975 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:06.073942 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:10.979085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:10.979027 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:14.372335 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:14.372304 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:15.807278 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:15.807241 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:16.908337 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:16.908305 4312 generic.go:356] "Generic (PLEG): container finished" podID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerID="d17b8c45bafea9589afc776a730c94d01ab05f8054ff4adda2f9afb88e81de45" exitCode=0 Nov 02 12:28:16.908730 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:16.908382 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" event={"ID":"ad378898-e30c-4945-9b5f-a310b62ec3b8","Type":"ContainerDied","Data":"d17b8c45bafea9589afc776a730c94d01ab05f8054ff4adda2f9afb88e81de45"} Nov 02 12:28:16.908899 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:16.908883 4312 scope.go:122] "RemoveContainer" containerID="d17b8c45bafea9589afc776a730c94d01ab05f8054ff4adda2f9afb88e81de45" Nov 02 12:28:16.910380 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:16.910357 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:28:16.910433 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:16.910424 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:28:16.910469 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:16.910456 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:28:17.915806 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:17.915772 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" event={"ID":"ad378898-e30c-4945-9b5f-a310b62ec3b8","Type":"ContainerStarted","Data":"4d415d30d13a1b416da0feb9da6cf1639c2238856f139bf7bf212bd30e776d69"} Nov 02 12:28:18.657978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:18.657946 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:28:18.658250 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:18.658098 4312 patch_prober.go:29] interesting pod/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 12:28:18.658400 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:18.658377 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 12:28:19.430649 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:19.430617 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:28:19.672594 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:19.671633 4312 patch_prober.go:29] interesting pod/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 12:28:19.672594 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:19.671689 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 12:28:20.643286 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:20.643242 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:20.668637 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:20.668600 4312 patch_prober.go:29] interesting pod/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 12:28:20.668923 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:20.668896 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 12:28:21.666522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:21.660784 4312 patch_prober.go:29] interesting pod/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 12:28:21.666522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:21.660844 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 12:28:22.667839 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:22.667804 4312 patch_prober.go:29] interesting pod/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Nov 02 12:28:22.668422 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:22.668388 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Nov 02 12:28:23.661538 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:23.661487 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:28:23.662487 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:23.662468 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:28:25.514366 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:25.514336 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:29.373469 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:28:29.373422 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:28:30.389045 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:30.389011 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:35.214762 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:35.214726 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:40.102309 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:40.102265 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:45.553341 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:45.553307 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:50.489820 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:50.489786 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:54.857669 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:54.857635 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g"] Nov 02 12:28:55.334827 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:55.334797 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:55.586813 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:55.586739 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:28:59.373001 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:28:59.372964 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:28:59.858367 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:28:59.858332 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" podUID="0bf4e46c-ad51-4527-8cfb-3c01f31da9d8" containerName="payload-processing" containerID="cri-o://81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8" gracePeriod=25 Nov 02 12:29:00.115145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.115086 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:29:00.131879 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.131848 4312 generic.go:356] "Generic (PLEG): container finished" podID="0bf4e46c-ad51-4527-8cfb-3c01f31da9d8" containerID="81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8" exitCode=0 Nov 02 12:29:00.132019 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.131892 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" Nov 02 12:29:00.132019 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.131930 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" event={"ID":"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8","Type":"ContainerDied","Data":"81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8"} Nov 02 12:29:00.132019 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.132000 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g" event={"ID":"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8","Type":"ContainerDied","Data":"9a7c94baff4ac3eee8dc59ee11266122ea36eba69ada6ea77c424689b8757402"} Nov 02 12:29:00.132185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.132025 4312 scope.go:122] "RemoveContainer" containerID="81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8" Nov 02 12:29:00.143653 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.143630 4312 scope.go:122] "RemoveContainer" containerID="81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8" Nov 02 12:29:00.144046 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:29:00.144023 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8\": container with ID starting with 81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8 not found: ID does not exist" containerID="81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8" Nov 02 12:29:00.144126 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.144057 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8"} err="failed to get container status \"81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8\": rpc error: code = NotFound desc = could not find container \"81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8\": container with ID starting with 81d168952127630ece5860413ade496bd6285119b109b09d580eb71ef34a84f8 not found: ID does not exist" Nov 02 12:29:00.184246 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.184217 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-plugins-config-volume\") pod \"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8\" (UID: \"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8\") " Nov 02 12:29:00.184357 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.184314 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-kube-api-access-cl2x7\" (UniqueName: \"kubernetes.io/projected/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-kube-api-access-cl2x7\") pod \"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8\" (UID: \"0bf4e46c-ad51-4527-8cfb-3c01f31da9d8\") " Nov 02 12:29:00.184589 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.184566 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-plugins-config-volume" pod "0bf4e46c-ad51-4527-8cfb-3c01f31da9d8" (UID: "0bf4e46c-ad51-4527-8cfb-3c01f31da9d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:29:00.184696 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.184683 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-plugins-config-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:00.186247 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.186225 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-kube-api-access-cl2x7" pod "0bf4e46c-ad51-4527-8cfb-3c01f31da9d8" (UID: "0bf4e46c-ad51-4527-8cfb-3c01f31da9d8"). InnerVolumeSpecName "kube-api-access-cl2x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:29:00.285472 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.285431 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl2x7\" (UniqueName: \"kubernetes.io/projected/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8-kube-api-access-cl2x7\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:00.293839 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.293810 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:00.552689 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.552661 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g"] Nov 02 12:29:00.556300 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:00.556271 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ccee868d-579cd68b9f-rt76g"] Nov 02 12:29:01.075368 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:01.075339 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf4e46c-ad51-4527-8cfb-3c01f31da9d8" path="/var/lib/kubelet/pods/0bf4e46c-ad51-4527-8cfb-3c01f31da9d8/volumes" Nov 02 12:29:05.186896 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:05.186860 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:06.823589 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:06.823563 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:10.078540 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:10.078486 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:14.110299 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.110266 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm"] Nov 02 12:29:14.110841 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.110486 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" podUID="01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb" containerName="maas-api" containerID="cri-o://00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778" gracePeriod=30 Nov 02 12:29:14.314807 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.314685 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:14.412339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.412305 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:29:14.524850 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.524817 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-metrics-certs\") pod \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " Nov 02 12:29:14.525050 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.524908 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls\") pod \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " Nov 02 12:29:14.525050 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.524999 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-kube-api-access-r5v9w\" (UniqueName: \"kubernetes.io/projected/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-kube-api-access-r5v9w\") pod \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " Nov 02 12:29:14.525185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.525094 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-openshift-service-ca\") pod \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\" (UID: \"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb\") " Nov 02 12:29:14.525827 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.525801 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-openshift-service-ca" pod "01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb" (UID: "01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:29:14.527777 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.527737 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-metrics-certs" pod "01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb" (UID: "01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:29:14.527877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.527827 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls" pod "01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb" (UID: "01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:29:14.528025 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.528004 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-kube-api-access-r5v9w" pod "01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb" (UID: "01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb"). InnerVolumeSpecName "kube-api-access-r5v9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:29:14.626842 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.626812 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-maas-api-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:14.626842 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.626841 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5v9w\" (UniqueName: \"kubernetes.io/projected/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-kube-api-access-r5v9w\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:14.627042 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.626851 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-openshift-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:14.627042 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.626860 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb-metrics-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:14.699925 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:14.699866 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:15.212097 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:15.212061 4312 generic.go:356] "Generic (PLEG): container finished" podID="01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb" containerID="00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778" exitCode=0 Nov 02 12:29:15.212646 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:15.212107 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" Nov 02 12:29:15.212646 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:15.212151 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" event={"ID":"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb","Type":"ContainerDied","Data":"00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778"} Nov 02 12:29:15.212646 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:15.212213 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm" event={"ID":"01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb","Type":"ContainerDied","Data":"823c8d0a6bc3b3c9537ee4db085caec88e6ae71d1e1416c0e44428b5d52b214d"} Nov 02 12:29:15.212646 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:15.212247 4312 scope.go:122] "RemoveContainer" containerID="00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778" Nov 02 12:29:15.221610 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:15.221592 4312 scope.go:122] "RemoveContainer" containerID="00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778" Nov 02 12:29:15.221859 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:29:15.221839 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778\": container with ID starting with 00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778 not found: ID does not exist" containerID="00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778" Nov 02 12:29:15.221940 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:15.221871 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778"} err="failed to get container status \"00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778\": rpc error: code = NotFound desc = could not find container \"00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778\": container with ID starting with 00b004105b6245ce5ca054ed0ddf396de07602de795483e611ec5428a9552778 not found: ID does not exist" Nov 02 12:29:15.377092 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:15.377027 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm"] Nov 02 12:29:15.427094 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:15.427070 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ccee868d-67f66957f9-jxpxm"] Nov 02 12:29:15.892603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:15.892554 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:16.169527 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:16.169483 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:16.417095 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:16.417063 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:17.005822 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:17.005792 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp"] Nov 02 12:29:17.006044 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:17.006020 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" containerID="cri-o://4d415d30d13a1b416da0feb9da6cf1639c2238856f139bf7bf212bd30e776d69" gracePeriod=30 Nov 02 12:29:17.075442 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:17.075408 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb" path="/var/lib/kubelet/pods/01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb/volumes" Nov 02 12:29:17.118532 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:17.118434 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg"] Nov 02 12:29:17.118716 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:17.118690 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" podUID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" containerName="istio-proxy" containerID="cri-o://a3ac1e6bd18de0b17e7b47bb276f5b5a3ffff6e80d2a052e6cd699bb59bde5eb" gracePeriod=30 Nov 02 12:29:17.159519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:17.159074 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:17.856380 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:17.856346 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:18.230522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.230474 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8"] Nov 02 12:29:18.231156 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.231138 4312 cpu_manager.go:414] "containerMap: removing container" podUID="01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb" containerName="maas-api" Nov 02 12:29:18.231200 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.231161 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb" containerName="maas-api" Nov 02 12:29:18.231235 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.231200 4312 cpu_manager.go:414] "containerMap: removing container" podUID="0bf4e46c-ad51-4527-8cfb-3c01f31da9d8" containerName="payload-processing" Nov 02 12:29:18.231235 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.231209 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf4e46c-ad51-4527-8cfb-3c01f31da9d8" containerName="payload-processing" Nov 02 12:29:18.231344 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.231333 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ec84fd-0767-4bf2-b0a4-3b4c6a6a0bbb" containerName="maas-api" Nov 02 12:29:18.231378 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.231351 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf4e46c-ad51-4527-8cfb-3c01f31da9d8" containerName="payload-processing" Nov 02 12:29:18.234878 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.234856 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.302406 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.302365 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8"] Nov 02 12:29:18.349412 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.349380 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-02304760-openshift-default-dockercfg-2hmvr\"" Nov 02 12:29:18.380642 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.380597 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-socket\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.380642 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.380644 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-podinfo\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.380897 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.380767 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-certs\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.380897 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.380801 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-envoy\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.380897 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.380826 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-data\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.381057 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.380938 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-credential-socket\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.381057 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.380981 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk257\" (UniqueName: \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-kube-api-access-vk257\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.381057 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.381020 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-token\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.381215 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.381095 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istiod-ca-cert\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.413364 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.413329 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:18.484607 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.484490 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-certs\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.484864 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.484842 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-envoy\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.484983 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.484968 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-data\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.485113 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.485087 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-credential-socket\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.485261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.485139 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk257\" (UniqueName: \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-kube-api-access-vk257\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.485261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.485166 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-token\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.485398 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.485291 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istiod-ca-cert\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.485398 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.485335 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-socket\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.485398 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.485371 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-podinfo\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.486568 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.486238 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-certs\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.486568 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.486491 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-socket\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.486891 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.486862 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istiod-ca-cert\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.487010 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.486986 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-credential-socket\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.487145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.487118 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-data\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.488621 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.488595 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-podinfo\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.492949 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.492922 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-envoy\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.547240 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.547204 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk257\" (UniqueName: \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-kube-api-access-vk257\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.547757 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.547727 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-token\") pod \"e2e-derive-02304760-openshift-default-5f76687f89-s72l8\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.551606 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.551584 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:18.664856 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.664803 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:18.982330 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.982308 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8"] Nov 02 12:29:18.984127 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:29:18.984096 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65fcfca_ab6f_45bf_80b0_82a4671bc391.slice/crio-f153c2a6b51bf5bb4c3c4efdd18a5905b24330a141f84bd3d49e43fd8b542982 WatchSource:0}: Error finding container f153c2a6b51bf5bb4c3c4efdd18a5905b24330a141f84bd3d49e43fd8b542982: Status 404 returned error can't find the container with id f153c2a6b51bf5bb4c3c4efdd18a5905b24330a141f84bd3d49e43fd8b542982 Nov 02 12:29:18.986433 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.986405 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:29:18.986530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.986465 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:29:18.986530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:18.986512 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:29:19.235886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:19.235803 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" event={"ID":"a65fcfca-ab6f-45bf-80b0-82a4671bc391","Type":"ContainerStarted","Data":"d4fd05424c1b0ef1a11a6376b8187debdef470dda9db2d18bd2e8b8e4e358a4f"} Nov 02 12:29:19.235886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:19.235849 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" event={"ID":"a65fcfca-ab6f-45bf-80b0-82a4671bc391","Type":"ContainerStarted","Data":"f153c2a6b51bf5bb4c3c4efdd18a5905b24330a141f84bd3d49e43fd8b542982"} Nov 02 12:29:19.356968 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:19.356914 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" podStartSLOduration=1.356901024 podStartE2EDuration="1.356901024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:29:19.354769666 +0000 UTC m=+1976.385005917" watchObservedRunningTime="2026-11-02 12:29:19.356901024 +0000 UTC m=+1976.387137274" Nov 02 12:29:19.551725 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:19.551645 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:20.557006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:20.556969 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:21.248536 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:21.248486 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:21.249507 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:21.249467 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:29:21.561099 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:21.561013 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:21.806463 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:21.806427 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:22.054206 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.054176 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:22.256300 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.256261 4312 generic.go:356] "Generic (PLEG): container finished" podID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerID="4d415d30d13a1b416da0feb9da6cf1639c2238856f139bf7bf212bd30e776d69" exitCode=0 Nov 02 12:29:22.256425 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.256330 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" event={"ID":"ad378898-e30c-4945-9b5f-a310b62ec3b8","Type":"ContainerDied","Data":"4d415d30d13a1b416da0feb9da6cf1639c2238856f139bf7bf212bd30e776d69"} Nov 02 12:29:22.256425 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.256383 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" event={"ID":"ad378898-e30c-4945-9b5f-a310b62ec3b8","Type":"ContainerDied","Data":"d65214561fc573829456c3f79040a1e12c71acfed140006f81460ce7a3bd2e58"} Nov 02 12:29:22.256425 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.256405 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d65214561fc573829456c3f79040a1e12c71acfed140006f81460ce7a3bd2e58" Nov 02 12:29:22.256425 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.256422 4312 scope.go:122] "RemoveContainer" containerID="d17b8c45bafea9589afc776a730c94d01ab05f8054ff4adda2f9afb88e81de45" Nov 02 12:29:22.258348 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.258328 4312 generic.go:356] "Generic (PLEG): container finished" podID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" containerID="a3ac1e6bd18de0b17e7b47bb276f5b5a3ffff6e80d2a052e6cd699bb59bde5eb" exitCode=0 Nov 02 12:29:22.258465 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.258406 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" event={"ID":"6f6435ad-c2f0-4586-a026-c4f57abcbb05","Type":"ContainerDied","Data":"a3ac1e6bd18de0b17e7b47bb276f5b5a3ffff6e80d2a052e6cd699bb59bde5eb"} Nov 02 12:29:22.263339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.263318 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:29:22.376728 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.376708 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:29:22.428776 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.428749 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-data\") pod \"ad378898-e30c-4945-9b5f-a310b62ec3b8\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " Nov 02 12:29:22.428899 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.428825 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-envoy\") pod \"ad378898-e30c-4945-9b5f-a310b62ec3b8\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " Nov 02 12:29:22.428944 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.428918 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad378898-e30c-4945-9b5f-a310b62ec3b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad378898-e30c-4945-9b5f-a310b62ec3b8-istiod-ca-cert\") pod \"ad378898-e30c-4945-9b5f-a310b62ec3b8\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " Nov 02 12:29:22.428980 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.428951 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-podinfo\") pod \"ad378898-e30c-4945-9b5f-a310b62ec3b8\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " Nov 02 12:29:22.429021 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429006 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-socket\") pod \"ad378898-e30c-4945-9b5f-a310b62ec3b8\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " Nov 02 12:29:22.429067 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429036 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-data" pod "ad378898-e30c-4945-9b5f-a310b62ec3b8" (UID: "ad378898-e30c-4945-9b5f-a310b62ec3b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:29:22.429067 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429046 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-token\" (UniqueName: \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-token\") pod \"ad378898-e30c-4945-9b5f-a310b62ec3b8\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " Nov 02 12:29:22.429208 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429189 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-kube-api-access-f7w5k\" (UniqueName: \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-kube-api-access-f7w5k\") pod \"ad378898-e30c-4945-9b5f-a310b62ec3b8\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " Nov 02 12:29:22.429290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429248 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-certs\") pod \"ad378898-e30c-4945-9b5f-a310b62ec3b8\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " Nov 02 12:29:22.429290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429278 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-credential-socket\") pod \"ad378898-e30c-4945-9b5f-a310b62ec3b8\" (UID: \"ad378898-e30c-4945-9b5f-a310b62ec3b8\") " Nov 02 12:29:22.429290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429283 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-socket" pod "ad378898-e30c-4945-9b5f-a310b62ec3b8" (UID: "ad378898-e30c-4945-9b5f-a310b62ec3b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:29:22.429458 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429305 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad378898-e30c-4945-9b5f-a310b62ec3b8-istiod-ca-cert" pod "ad378898-e30c-4945-9b5f-a310b62ec3b8" (UID: "ad378898-e30c-4945-9b5f-a310b62ec3b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:29:22.429931 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429559 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-certs" pod "ad378898-e30c-4945-9b5f-a310b62ec3b8" (UID: "ad378898-e30c-4945-9b5f-a310b62ec3b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:29:22.429931 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429663 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-credential-socket" pod "ad378898-e30c-4945-9b5f-a310b62ec3b8" (UID: "ad378898-e30c-4945-9b5f-a310b62ec3b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:29:22.429931 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429911 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.429931 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429931 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.430224 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429942 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.430224 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429953 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad378898-e30c-4945-9b5f-a310b62ec3b8-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.430224 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.429967 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.431311 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.431293 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-kube-api-access-f7w5k" pod "ad378898-e30c-4945-9b5f-a310b62ec3b8" (UID: "ad378898-e30c-4945-9b5f-a310b62ec3b8"). InnerVolumeSpecName "kube-api-access-f7w5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:29:22.431386 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.431342 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-token" pod "ad378898-e30c-4945-9b5f-a310b62ec3b8" (UID: "ad378898-e30c-4945-9b5f-a310b62ec3b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:29:22.431448 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.431439 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-podinfo" pod "ad378898-e30c-4945-9b5f-a310b62ec3b8" (UID: "ad378898-e30c-4945-9b5f-a310b62ec3b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:29:22.431519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.431458 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-envoy" pod "ad378898-e30c-4945-9b5f-a310b62ec3b8" (UID: "ad378898-e30c-4945-9b5f-a310b62ec3b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:29:22.530623 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.530595 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-envoy\") pod \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " Nov 02 12:29:22.530724 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.530627 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-credential-socket\") pod \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " Nov 02 12:29:22.530724 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.530690 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-kube-api-access-zmljj\" (UniqueName: \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-kube-api-access-zmljj\") pod \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " Nov 02 12:29:22.530724 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.530718 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-podinfo\") pod \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " Nov 02 12:29:22.530890 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.530741 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-data\") pod \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " Nov 02 12:29:22.530890 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.530767 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istiod-ca-cert\") pod \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " Nov 02 12:29:22.531520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531046 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-socket\") pod \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " Nov 02 12:29:22.531520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531118 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-data" pod "6f6435ad-c2f0-4586-a026-c4f57abcbb05" (UID: "6f6435ad-c2f0-4586-a026-c4f57abcbb05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:29:22.531520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531135 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istiod-ca-cert" pod "6f6435ad-c2f0-4586-a026-c4f57abcbb05" (UID: "6f6435ad-c2f0-4586-a026-c4f57abcbb05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:29:22.531520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531186 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-token\" (UniqueName: \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-token\") pod \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " Nov 02 12:29:22.531520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531217 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-certs\") pod \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\" (UID: \"6f6435ad-c2f0-4586-a026-c4f57abcbb05\") " Nov 02 12:29:22.531520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531218 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-credential-socket" pod "6f6435ad-c2f0-4586-a026-c4f57abcbb05" (UID: "6f6435ad-c2f0-4586-a026-c4f57abcbb05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:29:22.531520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531310 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-socket" pod "6f6435ad-c2f0-4586-a026-c4f57abcbb05" (UID: "6f6435ad-c2f0-4586-a026-c4f57abcbb05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:29:22.531520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531461 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-certs" pod "6f6435ad-c2f0-4586-a026-c4f57abcbb05" (UID: "6f6435ad-c2f0-4586-a026-c4f57abcbb05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:29:22.531978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531677 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.531978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531697 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.531978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531711 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.531978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531725 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.531978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531739 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.531978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531752 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.531978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531768 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.531978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531783 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7w5k\" (UniqueName: \"kubernetes.io/projected/ad378898-e30c-4945-9b5f-a310b62ec3b8-kube-api-access-f7w5k\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.531978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.531797 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.533089 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.533066 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-envoy" pod "6f6435ad-c2f0-4586-a026-c4f57abcbb05" (UID: "6f6435ad-c2f0-4586-a026-c4f57abcbb05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:29:22.533190 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.533167 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-token" pod "6f6435ad-c2f0-4586-a026-c4f57abcbb05" (UID: "6f6435ad-c2f0-4586-a026-c4f57abcbb05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:29:22.533420 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.533402 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-kube-api-access-zmljj" pod "6f6435ad-c2f0-4586-a026-c4f57abcbb05" (UID: "6f6435ad-c2f0-4586-a026-c4f57abcbb05"). InnerVolumeSpecName "kube-api-access-zmljj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:29:22.533583 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.533565 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-podinfo" pod "6f6435ad-c2f0-4586-a026-c4f57abcbb05" (UID: "6f6435ad-c2f0-4586-a026-c4f57abcbb05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:29:22.632579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.632525 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmljj\" (UniqueName: \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-kube-api-access-zmljj\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.632579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.632547 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.632579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.632556 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.632579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.632564 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f6435ad-c2f0-4586-a026-c4f57abcbb05-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:22.906444 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:22.906368 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:23.264799 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:23.264776 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp" Nov 02 12:29:23.266216 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:23.266195 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" Nov 02 12:29:23.266342 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:23.266249 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg" event={"ID":"6f6435ad-c2f0-4586-a026-c4f57abcbb05","Type":"ContainerDied","Data":"f9148dca53815911c37876fa467db994700671d510059164703b0cbb0eaa29f0"} Nov 02 12:29:23.266342 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:23.266314 4312 scope.go:122] "RemoveContainer" containerID="a3ac1e6bd18de0b17e7b47bb276f5b5a3ffff6e80d2a052e6cd699bb59bde5eb" Nov 02 12:29:23.545709 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:23.545653 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg"] Nov 02 12:29:23.549334 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:23.549312 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-gbqcg"] Nov 02 12:29:23.657572 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:23.657546 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp"] Nov 02 12:29:23.663438 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:23.663416 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ccee868d-openshift-default-5c76b6887c-ttszp"] Nov 02 12:29:24.818412 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:24.818363 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:25.063415 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:25.063355 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:25.078296 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:25.078218 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" path="/var/lib/kubelet/pods/6f6435ad-c2f0-4586-a026-c4f57abcbb05/volumes" Nov 02 12:29:25.078745 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:25.078730 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" path="/var/lib/kubelet/pods/ad378898-e30c-4945-9b5f-a310b62ec3b8/volumes" Nov 02 12:29:27.148898 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:27.148872 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:28.831004 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:28.830969 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:29.372168 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:29:29.372132 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:29:31.702656 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.702627 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7227-18ebed"] Nov 02 12:29:31.703063 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.703035 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" Nov 02 12:29:31.703063 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.703046 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" Nov 02 12:29:31.703063 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.703057 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" Nov 02 12:29:31.703063 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.703062 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" Nov 02 12:29:31.703207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.703073 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" containerName="istio-proxy" Nov 02 12:29:31.703207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.703079 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" containerName="istio-proxy" Nov 02 12:29:31.703207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.703165 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" Nov 02 12:29:31.703207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.703173 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad378898-e30c-4945-9b5f-a310b62ec3b8" containerName="istio-proxy" Nov 02 12:29:31.703207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.703180 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f6435ad-c2f0-4586-a026-c4f57abcbb05" containerName="istio-proxy" Nov 02 12:29:31.707956 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.707939 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-18ebed" Nov 02 12:29:31.728070 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.728048 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tpgf\" (UniqueName: \"kubernetes.io/projected/d727702d-a1a3-44c5-8044-12ed9c3d015a-kube-api-access-5tpgf\") pod \"test-curl-7227-18ebed\" (UID: \"d727702d-a1a3-44c5-8044-12ed9c3d015a\") " pod="opendatahub/test-curl-7227-18ebed" Nov 02 12:29:31.762413 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.762388 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5lqpk\"" Nov 02 12:29:31.765805 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.765782 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-18ebed"] Nov 02 12:29:31.828759 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.828731 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpgf\" (UniqueName: \"kubernetes.io/projected/d727702d-a1a3-44c5-8044-12ed9c3d015a-kube-api-access-5tpgf\") pod \"test-curl-7227-18ebed\" (UID: \"d727702d-a1a3-44c5-8044-12ed9c3d015a\") " pod="opendatahub/test-curl-7227-18ebed" Nov 02 12:29:31.889454 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:31.889432 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tpgf\" (UniqueName: \"kubernetes.io/projected/d727702d-a1a3-44c5-8044-12ed9c3d015a-kube-api-access-5tpgf\") pod \"test-curl-7227-18ebed\" (UID: \"d727702d-a1a3-44c5-8044-12ed9c3d015a\") " pod="opendatahub/test-curl-7227-18ebed" Nov 02 12:29:32.012761 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:32.012671 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:32.019657 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:32.019634 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-18ebed" Nov 02 12:29:32.393550 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:32.393518 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-18ebed"] Nov 02 12:29:32.395155 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:29:32.395114 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd727702d_a1a3_44c5_8044_12ed9c3d015a.slice/crio-f09b8b10f0885c4a3af8670b51ab925bf3eee12fdbe6f659c4c8ed7bccdf5fb5 WatchSource:0}: Error finding container f09b8b10f0885c4a3af8670b51ab925bf3eee12fdbe6f659c4c8ed7bccdf5fb5: Status 404 returned error can't find the container with id f09b8b10f0885c4a3af8670b51ab925bf3eee12fdbe6f659c4c8ed7bccdf5fb5 Nov 02 12:29:33.343341 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:33.343309 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-18ebed" event={"ID":"d727702d-a1a3-44c5-8044-12ed9c3d015a","Type":"ContainerStarted","Data":"6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5"} Nov 02 12:29:33.343758 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:33.343352 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-18ebed" event={"ID":"d727702d-a1a3-44c5-8044-12ed9c3d015a","Type":"ContainerStarted","Data":"f09b8b10f0885c4a3af8670b51ab925bf3eee12fdbe6f659c4c8ed7bccdf5fb5"} Nov 02 12:29:33.460203 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:33.460155 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7227-18ebed" podStartSLOduration=2.460141967 podStartE2EDuration="2.460141967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:29:33.458022815 +0000 UTC m=+1990.488259068" watchObservedRunningTime="2026-11-02 12:29:33.460141967 +0000 UTC m=+1990.490378216" Nov 02 12:29:34.879755 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:34.879722 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7227-18ebed"] Nov 02 12:29:34.884672 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:34.884643 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7227-18ebed"] Nov 02 12:29:35.035117 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.035086 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7227-77f79b"] Nov 02 12:29:35.035564 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.035549 4312 cpu_manager.go:414] "containerMap: removing container" podUID="d727702d-a1a3-44c5-8044-12ed9c3d015a" containerName="test-curl-7227-18ebed" Nov 02 12:29:35.035629 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.035565 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d727702d-a1a3-44c5-8044-12ed9c3d015a" containerName="test-curl-7227-18ebed" Nov 02 12:29:35.035668 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.035635 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="d727702d-a1a3-44c5-8044-12ed9c3d015a" containerName="test-curl-7227-18ebed" Nov 02 12:29:35.038834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.038809 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-77f79b" Nov 02 12:29:35.093030 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.093005 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-77f79b"] Nov 02 12:29:35.160690 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.160631 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzk6\" (UniqueName: \"kubernetes.io/projected/dc74d061-0626-4c88-9553-aaca95c7e392-kube-api-access-zbzk6\") pod \"test-curl-7227-77f79b\" (UID: \"dc74d061-0626-4c88-9553-aaca95c7e392\") " pod="opendatahub/test-curl-7227-77f79b" Nov 02 12:29:35.262057 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.262023 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzk6\" (UniqueName: \"kubernetes.io/projected/dc74d061-0626-4c88-9553-aaca95c7e392-kube-api-access-zbzk6\") pod \"test-curl-7227-77f79b\" (UID: \"dc74d061-0626-4c88-9553-aaca95c7e392\") " pod="opendatahub/test-curl-7227-77f79b" Nov 02 12:29:35.322745 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.322720 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzk6\" (UniqueName: \"kubernetes.io/projected/dc74d061-0626-4c88-9553-aaca95c7e392-kube-api-access-zbzk6\") pod \"test-curl-7227-77f79b\" (UID: \"dc74d061-0626-4c88-9553-aaca95c7e392\") " pod="opendatahub/test-curl-7227-77f79b" Nov 02 12:29:35.350408 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.350380 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-77f79b" Nov 02 12:29:35.354238 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.354214 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7227-18ebed" podUID="d727702d-a1a3-44c5-8044-12ed9c3d015a" containerName="test-curl-7227-18ebed" containerID="cri-o://6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5" gracePeriod=2 Nov 02 12:29:35.719372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:35.719347 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7227-77f79b"] Nov 02 12:29:35.721184 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:29:35.721157 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc74d061_0626_4c88_9553_aaca95c7e392.slice/crio-75ceef3481765cd883fdf93e8fe49c6c68c44e0e1876539053f4397223c98d9b WatchSource:0}: Error finding container 75ceef3481765cd883fdf93e8fe49c6c68c44e0e1876539053f4397223c98d9b: Status 404 returned error can't find the container with id 75ceef3481765cd883fdf93e8fe49c6c68c44e0e1876539053f4397223c98d9b Nov 02 12:29:36.361087 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:36.361053 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-77f79b" event={"ID":"dc74d061-0626-4c88-9553-aaca95c7e392","Type":"ContainerStarted","Data":"28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45"} Nov 02 12:29:36.361463 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:36.361098 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7227-77f79b" event={"ID":"dc74d061-0626-4c88-9553-aaca95c7e392","Type":"ContainerStarted","Data":"75ceef3481765cd883fdf93e8fe49c6c68c44e0e1876539053f4397223c98d9b"} Nov 02 12:29:36.475182 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:36.475129 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7227-77f79b" podStartSLOduration=2.475115302 podStartE2EDuration="2.475115302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:29:36.473137536 +0000 UTC m=+1993.503373788" watchObservedRunningTime="2026-11-02 12:29:36.475115302 +0000 UTC m=+1993.505351551" Nov 02 12:29:36.898626 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:36.898592 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:37.584930 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:37.584907 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-18ebed" Nov 02 12:29:37.684881 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:37.684856 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d727702d-a1a3-44c5-8044-12ed9c3d015a-kube-api-access-5tpgf\" (UniqueName: \"kubernetes.io/projected/d727702d-a1a3-44c5-8044-12ed9c3d015a-kube-api-access-5tpgf\") pod \"d727702d-a1a3-44c5-8044-12ed9c3d015a\" (UID: \"d727702d-a1a3-44c5-8044-12ed9c3d015a\") " Nov 02 12:29:37.686945 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:37.686919 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d727702d-a1a3-44c5-8044-12ed9c3d015a-kube-api-access-5tpgf" pod "d727702d-a1a3-44c5-8044-12ed9c3d015a" (UID: "d727702d-a1a3-44c5-8044-12ed9c3d015a"). InnerVolumeSpecName "kube-api-access-5tpgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:29:37.786086 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:37.786059 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tpgf\" (UniqueName: \"kubernetes.io/projected/d727702d-a1a3-44c5-8044-12ed9c3d015a-kube-api-access-5tpgf\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:37.915531 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:37.915505 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7227-77f79b"] Nov 02 12:29:37.938022 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:37.937955 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7227-77f79b"] Nov 02 12:29:38.372447 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:38.372363 4312 generic.go:356] "Generic (PLEG): container finished" podID="d727702d-a1a3-44c5-8044-12ed9c3d015a" containerID="6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5" exitCode=137 Nov 02 12:29:38.372447 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:38.372420 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-18ebed" Nov 02 12:29:38.372663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:38.372461 4312 scope.go:122] "RemoveContainer" containerID="6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5" Nov 02 12:29:38.372797 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:38.372770 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7227-77f79b" podUID="dc74d061-0626-4c88-9553-aaca95c7e392" containerName="test-curl-7227-77f79b" containerID="cri-o://28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45" gracePeriod=2 Nov 02 12:29:38.381162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:38.381143 4312 scope.go:122] "RemoveContainer" containerID="6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5" Nov 02 12:29:38.381437 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:29:38.381419 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5\": container with ID starting with 6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5 not found: ID does not exist" containerID="6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5" Nov 02 12:29:38.381486 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:38.381445 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5"} err="failed to get container status \"6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5\": rpc error: code = NotFound desc = could not find container \"6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5\": container with ID starting with 6e1f737ec693b06df7387980f5e3c46397645cdef17093b4c92577a8a3fc66c5 not found: ID does not exist" Nov 02 12:29:38.423709 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:29:38.423678 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-77f79b\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="dc74d061-0626-4c88-9553-aaca95c7e392" pod="opendatahub/test-curl-7227-77f79b" Nov 02 12:29:38.474465 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:29:38.474438 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-18ebed\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="d727702d-a1a3-44c5-8044-12ed9c3d015a" pod="opendatahub/test-curl-7227-18ebed" Nov 02 12:29:38.525065 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:29:38.525041 4312 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7227-77f79b\" is forbidden: User \"system:node:ip-10-0-141-170.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-170.us-west-1.compute.internal' and this object" podUID="dc74d061-0626-4c88-9553-aaca95c7e392" pod="opendatahub/test-curl-7227-77f79b" Nov 02 12:29:39.075509 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:39.075474 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d727702d-a1a3-44c5-8044-12ed9c3d015a" path="/var/lib/kubelet/pods/d727702d-a1a3-44c5-8044-12ed9c3d015a/volumes" Nov 02 12:29:40.575670 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:40.575649 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-77f79b" Nov 02 12:29:40.714145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:40.714120 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc74d061-0626-4c88-9553-aaca95c7e392-kube-api-access-zbzk6\" (UniqueName: \"kubernetes.io/projected/dc74d061-0626-4c88-9553-aaca95c7e392-kube-api-access-zbzk6\") pod \"dc74d061-0626-4c88-9553-aaca95c7e392\" (UID: \"dc74d061-0626-4c88-9553-aaca95c7e392\") " Nov 02 12:29:40.716057 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:40.716032 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc74d061-0626-4c88-9553-aaca95c7e392-kube-api-access-zbzk6" pod "dc74d061-0626-4c88-9553-aaca95c7e392" (UID: "dc74d061-0626-4c88-9553-aaca95c7e392"). InnerVolumeSpecName "kube-api-access-zbzk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:29:40.815934 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:40.815910 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzk6\" (UniqueName: \"kubernetes.io/projected/dc74d061-0626-4c88-9553-aaca95c7e392-kube-api-access-zbzk6\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:29:41.075414 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:41.075348 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc74d061-0626-4c88-9553-aaca95c7e392" path="/var/lib/kubelet/pods/dc74d061-0626-4c88-9553-aaca95c7e392/volumes" Nov 02 12:29:41.390597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:41.390531 4312 generic.go:356] "Generic (PLEG): container finished" podID="dc74d061-0626-4c88-9553-aaca95c7e392" containerID="28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45" exitCode=137 Nov 02 12:29:41.390597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:41.390577 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7227-77f79b" Nov 02 12:29:41.390597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:41.390598 4312 scope.go:122] "RemoveContainer" containerID="28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45" Nov 02 12:29:41.398972 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:41.398955 4312 scope.go:122] "RemoveContainer" containerID="28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45" Nov 02 12:29:41.399240 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:29:41.399214 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45\": container with ID starting with 28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45 not found: ID does not exist" containerID="28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45" Nov 02 12:29:41.399290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:41.399250 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45"} err="failed to get container status \"28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45\": rpc error: code = NotFound desc = could not find container \"28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45\": container with ID starting with 28d9945f0e7b00aded55799fac27721f08e167ecfbc13aa5aae702fe3844eb45 not found: ID does not exist" Nov 02 12:29:41.801145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:41.801076 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:46.738793 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:46.738710 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:49.907570 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:49.907533 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx"] Nov 02 12:29:49.907984 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:49.907971 4312 cpu_manager.go:414] "containerMap: removing container" podUID="dc74d061-0626-4c88-9553-aaca95c7e392" containerName="test-curl-7227-77f79b" Nov 02 12:29:49.908028 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:49.907984 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc74d061-0626-4c88-9553-aaca95c7e392" containerName="test-curl-7227-77f79b" Nov 02 12:29:49.908072 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:49.908063 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc74d061-0626-4c88-9553-aaca95c7e392" containerName="test-curl-7227-77f79b" Nov 02 12:29:49.911234 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:49.911215 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:49.963324 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:49.963299 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-03c4e8-openshift-default-dockercfg-8bvrq\"" Nov 02 12:29:50.015316 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.015290 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx"] Nov 02 12:29:50.050598 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.050568 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-data\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.050723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.050644 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6831a2f5-a9b7-4c08-a825-4c80554c601c-istiod-ca-cert\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.050795 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.050749 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-podinfo\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.050853 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.050807 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.050910 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.050862 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-envoy\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.050985 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.050965 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-token\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.051034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.050994 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-credential-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.051034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.051028 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b7ld\" (UniqueName: \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-kube-api-access-7b7ld\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.051105 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.051086 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-certs\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.152589 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.152567 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-token\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.152721 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.152595 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-credential-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.152721 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.152625 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b7ld\" (UniqueName: \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-kube-api-access-7b7ld\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.152721 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.152650 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-certs\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.152721 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.152677 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-data\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.152721 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.152713 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6831a2f5-a9b7-4c08-a825-4c80554c601c-istiod-ca-cert\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.152974 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.152762 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-podinfo\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.152974 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.152813 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.152974 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.152846 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-envoy\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.153417 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.153373 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-credential-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.153628 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.153524 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-data\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.153723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.153611 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-certs\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.153861 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.153830 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6831a2f5-a9b7-4c08-a825-4c80554c601c-istiod-ca-cert\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.156186 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.156159 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-socket\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.162754 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.162727 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-envoy\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.163162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.163135 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-podinfo\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.212868 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.212847 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-token\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.213887 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.213864 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b7ld\" (UniqueName: \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-kube-api-access-7b7ld\") pod \"e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.223350 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.223325 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:50.651700 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.651673 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx"] Nov 02 12:29:50.653641 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:29:50.653606 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6831a2f5_a9b7_4c08_a825_4c80554c601c.slice/crio-7de8fc932e61431c6d58799bfef78a80fe83d871436d4c7628eb2844724dbe27 WatchSource:0}: Error finding container 7de8fc932e61431c6d58799bfef78a80fe83d871436d4c7628eb2844724dbe27: Status 404 returned error can't find the container with id 7de8fc932e61431c6d58799bfef78a80fe83d871436d4c7628eb2844724dbe27 Nov 02 12:29:50.656131 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.656097 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:29:50.656223 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.656164 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:29:50.656223 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:50.656196 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:29:51.473541 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:51.469292 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" event={"ID":"6831a2f5-a9b7-4c08-a825-4c80554c601c","Type":"ContainerStarted","Data":"876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438"} Nov 02 12:29:51.473541 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:51.469365 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" event={"ID":"6831a2f5-a9b7-4c08-a825-4c80554c601c","Type":"ContainerStarted","Data":"7de8fc932e61431c6d58799bfef78a80fe83d871436d4c7628eb2844724dbe27"} Nov 02 12:29:51.640614 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:51.640561 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" podStartSLOduration=2.640545259 podStartE2EDuration="2.640545259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:29:51.589415588 +0000 UTC m=+2008.619651839" watchObservedRunningTime="2026-11-02 12:29:51.640545259 +0000 UTC m=+2008.670781508" Nov 02 12:29:51.882623 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:51.882533 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:52.223566 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:52.223532 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:52.225369 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:52.225336 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 12:29:52.225529 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:52.225381 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" podUID="6831a2f5-a9b7-4c08-a825-4c80554c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 12:29:53.224128 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:53.224091 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 12:29:53.224556 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:53.224145 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" podUID="6831a2f5-a9b7-4c08-a825-4c80554c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 12:29:54.223814 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:54.223773 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 12:29:54.224000 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:54.223830 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" podUID="6831a2f5-a9b7-4c08-a825-4c80554c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 12:29:55.223998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:55.223966 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 12:29:55.224389 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:55.224019 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" podUID="6831a2f5-a9b7-4c08-a825-4c80554c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 12:29:55.607086 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:55.607009 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-b-3a78b4_payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr_91b75ed7-022a-4d07-9d7b-a699d52d087a/payload-processing/0.log" Nov 02 12:29:56.223936 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:56.223901 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 12:29:56.224131 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:56.223957 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" podUID="6831a2f5-a9b7-4c08-a825-4c80554c601c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 12:29:56.730843 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:56.730794 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:56.977808 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:56.977779 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:57.226426 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:57.226385 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:29:57.408186 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:57.408157 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:57.408882 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:57.408849 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:57.409732 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:29:57.409707 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:29:59.372674 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:29:59.372642 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:30:00.182770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.182740 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl"] Nov 02 12:30:00.186575 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.186559 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" Nov 02 12:30:00.239388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.239356 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nbw2z\"" Nov 02 12:30:00.282717 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.282686 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d59h\" (UniqueName: \"kubernetes.io/projected/a6125f4d-efec-4765-87e2-64d992685019-kube-api-access-8d59h\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl\" (UID: \"a6125f4d-efec-4765-87e2-64d992685019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" Nov 02 12:30:00.284006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.283976 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6125f4d-efec-4765-87e2-64d992685019-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl\" (UID: \"a6125f4d-efec-4765-87e2-64d992685019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" Nov 02 12:30:00.310605 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.310576 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl"] Nov 02 12:30:00.385120 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.385035 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6125f4d-efec-4765-87e2-64d992685019-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl\" (UID: \"a6125f4d-efec-4765-87e2-64d992685019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" Nov 02 12:30:00.385469 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.385265 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d59h\" (UniqueName: \"kubernetes.io/projected/a6125f4d-efec-4765-87e2-64d992685019-kube-api-access-8d59h\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl\" (UID: \"a6125f4d-efec-4765-87e2-64d992685019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" Nov 02 12:30:00.385652 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.385633 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6125f4d-efec-4765-87e2-64d992685019-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl\" (UID: \"a6125f4d-efec-4765-87e2-64d992685019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" Nov 02 12:30:00.445957 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.445900 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d59h\" (UniqueName: \"kubernetes.io/projected/a6125f4d-efec-4765-87e2-64d992685019-kube-api-access-8d59h\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl\" (UID: \"a6125f4d-efec-4765-87e2-64d992685019\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" Nov 02 12:30:00.500258 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.500234 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" Nov 02 12:30:00.876225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:00.876189 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl"] Nov 02 12:30:00.877787 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:30:00.877757 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6125f4d_efec_4765_87e2_64d992685019.slice/crio-16b437180079ae1d111388454833fc48da322b05fb171e3f122ec72f441715c5 WatchSource:0}: Error finding container 16b437180079ae1d111388454833fc48da322b05fb171e3f122ec72f441715c5: Status 404 returned error can't find the container with id 16b437180079ae1d111388454833fc48da322b05fb171e3f122ec72f441715c5 Nov 02 12:30:01.521821 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.521791 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94"] Nov 02 12:30:01.525728 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.525706 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:01.528567 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.528539 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" event={"ID":"a6125f4d-efec-4765-87e2-64d992685019","Type":"ContainerStarted","Data":"b027d8ac91a4010849a8c2ffb1332cc38699049511c9fc7d9e56569bf5458fc7"} Nov 02 12:30:01.528700 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.528581 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" event={"ID":"a6125f4d-efec-4765-87e2-64d992685019","Type":"ContainerStarted","Data":"16b437180079ae1d111388454833fc48da322b05fb171e3f122ec72f441715c5"} Nov 02 12:30:01.578049 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.578026 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 12:30:01.597233 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.597207 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba84ecdc-r2c94\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:01.597356 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.597300 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba84ecdc-r2c94\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:01.597511 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.597361 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgxnk\" (UniqueName: \"kubernetes.io/projected/de26b59a-f1d3-43a7-b504-7ccd797e80a3-kube-api-access-bgxnk\") pod \"maas-api-revoke-sub-ba84ecdc-r2c94\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:01.630078 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.630051 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94"] Nov 02 12:30:01.698054 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.698026 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba84ecdc-r2c94\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:01.698147 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.698059 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgxnk\" (UniqueName: \"kubernetes.io/projected/de26b59a-f1d3-43a7-b504-7ccd797e80a3-kube-api-access-bgxnk\") pod \"maas-api-revoke-sub-ba84ecdc-r2c94\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:01.698147 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.698142 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba84ecdc-r2c94\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:01.698747 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.698726 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba84ecdc-r2c94\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:01.700153 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.700134 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba84ecdc-r2c94\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:01.754534 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.754468 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" podStartSLOduration=1.754456721 podStartE2EDuration="1.754456721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:01.754111869 +0000 UTC m=+2018.784348120" watchObservedRunningTime="2026-11-02 12:30:01.754456721 +0000 UTC m=+2018.784692970" Nov 02 12:30:01.760887 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.760868 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgxnk\" (UniqueName: \"kubernetes.io/projected/de26b59a-f1d3-43a7-b504-7ccd797e80a3-kube-api-access-bgxnk\") pod \"maas-api-revoke-sub-ba84ecdc-r2c94\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:01.830453 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.830377 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:01.839349 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:01.839331 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:02.212695 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:02.212669 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94"] Nov 02 12:30:02.214300 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:30:02.214271 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde26b59a_f1d3_43a7_b504_7ccd797e80a3.slice/crio-0e7421d7ca3e3281b5516cb930158d90adc74b29a5b870bb32ac2b32db0443ce WatchSource:0}: Error finding container 0e7421d7ca3e3281b5516cb930158d90adc74b29a5b870bb32ac2b32db0443ce: Status 404 returned error can't find the container with id 0e7421d7ca3e3281b5516cb930158d90adc74b29a5b870bb32ac2b32db0443ce Nov 02 12:30:02.534676 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:02.534614 4312 generic.go:356] "Generic (PLEG): container finished" podID="a6125f4d-efec-4765-87e2-64d992685019" containerID="b027d8ac91a4010849a8c2ffb1332cc38699049511c9fc7d9e56569bf5458fc7" exitCode=60 Nov 02 12:30:02.535040 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:02.534689 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" event={"ID":"a6125f4d-efec-4765-87e2-64d992685019","Type":"ContainerDied","Data":"b027d8ac91a4010849a8c2ffb1332cc38699049511c9fc7d9e56569bf5458fc7"} Nov 02 12:30:02.535040 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:02.534943 4312 scope.go:122] "RemoveContainer" containerID="b027d8ac91a4010849a8c2ffb1332cc38699049511c9fc7d9e56569bf5458fc7" Nov 02 12:30:02.536352 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:02.536331 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" event={"ID":"de26b59a-f1d3-43a7-b504-7ccd797e80a3","Type":"ContainerStarted","Data":"0b1932931eff7ceeaca49b592ba330f66b558c420db734cbb3bcc2fb0c0f179b"} Nov 02 12:30:02.536442 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:02.536376 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" event={"ID":"de26b59a-f1d3-43a7-b504-7ccd797e80a3","Type":"ContainerStarted","Data":"0e7421d7ca3e3281b5516cb930158d90adc74b29a5b870bb32ac2b32db0443ce"} Nov 02 12:30:02.765741 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:02.765695 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" podStartSLOduration=1.765681835 podStartE2EDuration="1.765681835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:02.763247432 +0000 UTC m=+2019.793483681" watchObservedRunningTime="2026-11-02 12:30:02.765681835 +0000 UTC m=+2019.795918123" Nov 02 12:30:03.543794 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:03.543727 4312 generic.go:356] "Generic (PLEG): container finished" podID="a6125f4d-efec-4765-87e2-64d992685019" containerID="60ad6434a97c996d4b171d91c02afc428fce9af6e7bf6ecb3c5bb81899d7493b" exitCode=60 Nov 02 12:30:03.544202 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:03.543808 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" event={"ID":"a6125f4d-efec-4765-87e2-64d992685019","Type":"ContainerDied","Data":"60ad6434a97c996d4b171d91c02afc428fce9af6e7bf6ecb3c5bb81899d7493b"} Nov 02 12:30:03.544202 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:03.543856 4312 scope.go:122] "RemoveContainer" containerID="b027d8ac91a4010849a8c2ffb1332cc38699049511c9fc7d9e56569bf5458fc7" Nov 02 12:30:03.544202 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:03.543996 4312 scope.go:122] "RemoveContainer" containerID="60ad6434a97c996d4b171d91c02afc428fce9af6e7bf6ecb3c5bb81899d7493b" Nov 02 12:30:03.544344 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:30:03.544240 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl_odh-ai-gateway-infra(a6125f4d-efec-4765-87e2-64d992685019)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" podUID="a6125f4d-efec-4765-87e2-64d992685019" Nov 02 12:30:03.545367 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:03.545339 4312 generic.go:356] "Generic (PLEG): container finished" podID="de26b59a-f1d3-43a7-b504-7ccd797e80a3" containerID="0b1932931eff7ceeaca49b592ba330f66b558c420db734cbb3bcc2fb0c0f179b" exitCode=0 Nov 02 12:30:03.545474 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:03.545387 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" event={"ID":"de26b59a-f1d3-43a7-b504-7ccd797e80a3","Type":"ContainerDied","Data":"0b1932931eff7ceeaca49b592ba330f66b558c420db734cbb3bcc2fb0c0f179b"} Nov 02 12:30:04.553268 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:04.553239 4312 scope.go:122] "RemoveContainer" containerID="60ad6434a97c996d4b171d91c02afc428fce9af6e7bf6ecb3c5bb81899d7493b" Nov 02 12:30:04.553705 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:30:04.553529 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl_odh-ai-gateway-infra(a6125f4d-efec-4765-87e2-64d992685019)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" podUID="a6125f4d-efec-4765-87e2-64d992685019" Nov 02 12:30:04.701290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:04.701270 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:04.830940 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:04.830837 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-xks-ca\") pod \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " Nov 02 12:30:04.830940 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:04.830934 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de26b59a-f1d3-43a7-b504-7ccd797e80a3-kube-api-access-bgxnk\" (UniqueName: \"kubernetes.io/projected/de26b59a-f1d3-43a7-b504-7ccd797e80a3-kube-api-access-bgxnk\") pod \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " Nov 02 12:30:04.831179 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:04.830966 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-openshift-ca\") pod \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\" (UID: \"de26b59a-f1d3-43a7-b504-7ccd797e80a3\") " Nov 02 12:30:04.831814 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:04.831781 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-openshift-ca" pod "de26b59a-f1d3-43a7-b504-7ccd797e80a3" (UID: "de26b59a-f1d3-43a7-b504-7ccd797e80a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:30:04.836477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:04.836453 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-xks-ca" pod "de26b59a-f1d3-43a7-b504-7ccd797e80a3" (UID: "de26b59a-f1d3-43a7-b504-7ccd797e80a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:30:04.852030 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:04.845873 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de26b59a-f1d3-43a7-b504-7ccd797e80a3-kube-api-access-bgxnk" pod "de26b59a-f1d3-43a7-b504-7ccd797e80a3" (UID: "de26b59a-f1d3-43a7-b504-7ccd797e80a3"). InnerVolumeSpecName "kube-api-access-bgxnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:30:04.932377 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:04.932338 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-xks-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:04.932377 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:04.932370 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgxnk\" (UniqueName: \"kubernetes.io/projected/de26b59a-f1d3-43a7-b504-7ccd797e80a3-kube-api-access-bgxnk\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:04.932377 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:04.932382 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de26b59a-f1d3-43a7-b504-7ccd797e80a3-maas-api-openshift-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:05.561286 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:05.561252 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" event={"ID":"de26b59a-f1d3-43a7-b504-7ccd797e80a3","Type":"ContainerDied","Data":"0e7421d7ca3e3281b5516cb930158d90adc74b29a5b870bb32ac2b32db0443ce"} Nov 02 12:30:05.561286 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:05.561277 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94" Nov 02 12:30:05.561774 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:05.561337 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e7421d7ca3e3281b5516cb930158d90adc74b29a5b870bb32ac2b32db0443ce" Nov 02 12:30:06.801217 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:06.801178 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:09.168912 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:09.168871 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:09.418644 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:09.418609 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:10.488986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.488951 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg"] Nov 02 12:30:10.489418 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.489405 4312 cpu_manager.go:414] "containerMap: removing container" podUID="de26b59a-f1d3-43a7-b504-7ccd797e80a3" containerName="revoke-keys" Nov 02 12:30:10.489491 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.489419 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de26b59a-f1d3-43a7-b504-7ccd797e80a3" containerName="revoke-keys" Nov 02 12:30:10.489549 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.489524 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="de26b59a-f1d3-43a7-b504-7ccd797e80a3" containerName="revoke-keys" Nov 02 12:30:10.492952 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.492934 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:10.545109 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.545083 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 12:30:10.594227 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.594204 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqmzp\" (UniqueName: \"kubernetes.io/projected/da3d444a-4cc6-41ea-945f-8e066d221bce-kube-api-access-fqmzp\") pod \"maas-api-revoke-sub-3ea176dd-725kg\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:10.594397 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.594280 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ea176dd-725kg\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:10.594478 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.594395 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ea176dd-725kg\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:10.597485 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.597465 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg"] Nov 02 12:30:10.695937 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.695903 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqmzp\" (UniqueName: \"kubernetes.io/projected/da3d444a-4cc6-41ea-945f-8e066d221bce-kube-api-access-fqmzp\") pod \"maas-api-revoke-sub-3ea176dd-725kg\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:10.696125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.695980 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ea176dd-725kg\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:10.696125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.696009 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ea176dd-725kg\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:10.696711 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.696686 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ea176dd-725kg\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:10.698349 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.698327 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ea176dd-725kg\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:10.757183 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.757109 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqmzp\" (UniqueName: \"kubernetes.io/projected/da3d444a-4cc6-41ea-945f-8e066d221bce-kube-api-access-fqmzp\") pod \"maas-api-revoke-sub-3ea176dd-725kg\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:10.806022 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:10.805994 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:11.179867 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:11.179843 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg"] Nov 02 12:30:11.182389 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:30:11.182363 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3d444a_4cc6_41ea_945f_8e066d221bce.slice/crio-ac14018d00b58235c6f83338e67b9eb167582938097cdf566b04a52e83a3a3d4 WatchSource:0}: Error finding container ac14018d00b58235c6f83338e67b9eb167582938097cdf566b04a52e83a3a3d4: Status 404 returned error can't find the container with id ac14018d00b58235c6f83338e67b9eb167582938097cdf566b04a52e83a3a3d4 Nov 02 12:30:11.596372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:11.596340 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" event={"ID":"da3d444a-4cc6-41ea-945f-8e066d221bce","Type":"ContainerStarted","Data":"fa9fc0e7fd78b3ee5f61935289f228337055773eace90afa1538028d52071bdf"} Nov 02 12:30:11.596698 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:11.596399 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" event={"ID":"da3d444a-4cc6-41ea-945f-8e066d221bce","Type":"ContainerStarted","Data":"ac14018d00b58235c6f83338e67b9eb167582938097cdf566b04a52e83a3a3d4"} Nov 02 12:30:11.713351 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:11.713217 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" podStartSLOduration=1.713199861 podStartE2EDuration="1.713199861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:11.710714732 +0000 UTC m=+2028.740950983" watchObservedRunningTime="2026-11-02 12:30:11.713199861 +0000 UTC m=+2028.743436110" Nov 02 12:30:11.773228 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:11.773194 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:12.603966 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:12.603925 4312 generic.go:356] "Generic (PLEG): container finished" podID="da3d444a-4cc6-41ea-945f-8e066d221bce" containerID="fa9fc0e7fd78b3ee5f61935289f228337055773eace90afa1538028d52071bdf" exitCode=0 Nov 02 12:30:12.604368 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:12.603999 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" event={"ID":"da3d444a-4cc6-41ea-945f-8e066d221bce","Type":"ContainerDied","Data":"fa9fc0e7fd78b3ee5f61935289f228337055773eace90afa1538028d52071bdf"} Nov 02 12:30:13.738647 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:13.738624 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:13.826638 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:13.826615 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3d444a-4cc6-41ea-945f-8e066d221bce-kube-api-access-fqmzp\" (UniqueName: \"kubernetes.io/projected/da3d444a-4cc6-41ea-945f-8e066d221bce-kube-api-access-fqmzp\") pod \"da3d444a-4cc6-41ea-945f-8e066d221bce\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " Nov 02 12:30:13.826740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:13.826728 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-openshift-ca\") pod \"da3d444a-4cc6-41ea-945f-8e066d221bce\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " Nov 02 12:30:13.826794 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:13.826766 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-xks-ca\") pod \"da3d444a-4cc6-41ea-945f-8e066d221bce\" (UID: \"da3d444a-4cc6-41ea-945f-8e066d221bce\") " Nov 02 12:30:13.827124 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:13.827098 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-openshift-ca" pod "da3d444a-4cc6-41ea-945f-8e066d221bce" (UID: "da3d444a-4cc6-41ea-945f-8e066d221bce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:30:13.828661 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:13.828641 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3d444a-4cc6-41ea-945f-8e066d221bce-kube-api-access-fqmzp" pod "da3d444a-4cc6-41ea-945f-8e066d221bce" (UID: "da3d444a-4cc6-41ea-945f-8e066d221bce"). InnerVolumeSpecName "kube-api-access-fqmzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:30:13.828790 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:13.828768 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-xks-ca" pod "da3d444a-4cc6-41ea-945f-8e066d221bce" (UID: "da3d444a-4cc6-41ea-945f-8e066d221bce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:30:13.928202 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:13.928174 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqmzp\" (UniqueName: \"kubernetes.io/projected/da3d444a-4cc6-41ea-945f-8e066d221bce-kube-api-access-fqmzp\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:13.928202 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:13.928199 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-openshift-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:13.928361 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:13.928210 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da3d444a-4cc6-41ea-945f-8e066d221bce-maas-api-xks-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:14.616301 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:14.616275 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" Nov 02 12:30:14.616469 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:14.616269 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg" event={"ID":"da3d444a-4cc6-41ea-945f-8e066d221bce","Type":"ContainerDied","Data":"ac14018d00b58235c6f83338e67b9eb167582938097cdf566b04a52e83a3a3d4"} Nov 02 12:30:14.616469 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:14.616396 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac14018d00b58235c6f83338e67b9eb167582938097cdf566b04a52e83a3a3d4" Nov 02 12:30:15.042682 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:15.042650 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:16.700347 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:16.700311 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:17.068822 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:17.068750 4312 scope.go:122] "RemoveContainer" containerID="60ad6434a97c996d4b171d91c02afc428fce9af6e7bf6ecb3c5bb81899d7493b" Nov 02 12:30:17.634192 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:17.634110 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" event={"ID":"a6125f4d-efec-4765-87e2-64d992685019","Type":"ContainerStarted","Data":"8fecfc6d27300c2063807e9c8237bcc7f855356bc7d9fe3fb407e6ee8f4a27f2"} Nov 02 12:30:18.641318 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:18.641284 4312 generic.go:356] "Generic (PLEG): container finished" podID="a6125f4d-efec-4765-87e2-64d992685019" containerID="8fecfc6d27300c2063807e9c8237bcc7f855356bc7d9fe3fb407e6ee8f4a27f2" exitCode=60 Nov 02 12:30:18.641318 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:18.641318 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" event={"ID":"a6125f4d-efec-4765-87e2-64d992685019","Type":"ContainerDied","Data":"8fecfc6d27300c2063807e9c8237bcc7f855356bc7d9fe3fb407e6ee8f4a27f2"} Nov 02 12:30:18.641801 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:18.641367 4312 scope.go:122] "RemoveContainer" containerID="60ad6434a97c996d4b171d91c02afc428fce9af6e7bf6ecb3c5bb81899d7493b" Nov 02 12:30:18.641801 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:18.641738 4312 scope.go:122] "RemoveContainer" containerID="8fecfc6d27300c2063807e9c8237bcc7f855356bc7d9fe3fb407e6ee8f4a27f2" Nov 02 12:30:18.641959 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:30:18.641937 4312 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl_odh-ai-gateway-infra(a6125f4d-efec-4765-87e2-64d992685019)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" podUID="a6125f4d-efec-4765-87e2-64d992685019" Nov 02 12:30:18.762107 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:18.762081 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl"] Nov 02 12:30:19.794783 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:19.794761 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" Nov 02 12:30:19.984242 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:19.984215 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6125f4d-efec-4765-87e2-64d992685019-kube-api-access-8d59h\" (UniqueName: \"kubernetes.io/projected/a6125f4d-efec-4765-87e2-64d992685019-kube-api-access-8d59h\") pod \"a6125f4d-efec-4765-87e2-64d992685019\" (UID: \"a6125f4d-efec-4765-87e2-64d992685019\") " Nov 02 12:30:19.984386 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:19.984268 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6125f4d-efec-4765-87e2-64d992685019-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6125f4d-efec-4765-87e2-64d992685019-maas-api-service-ca\") pod \"a6125f4d-efec-4765-87e2-64d992685019\" (UID: \"a6125f4d-efec-4765-87e2-64d992685019\") " Nov 02 12:30:19.984729 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:19.984698 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6125f4d-efec-4765-87e2-64d992685019-maas-api-service-ca" pod "a6125f4d-efec-4765-87e2-64d992685019" (UID: "a6125f4d-efec-4765-87e2-64d992685019"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:30:19.984837 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:19.984798 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6125f4d-efec-4765-87e2-64d992685019-maas-api-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:19.986334 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:19.986307 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6125f4d-efec-4765-87e2-64d992685019-kube-api-access-8d59h" pod "a6125f4d-efec-4765-87e2-64d992685019" (UID: "a6125f4d-efec-4765-87e2-64d992685019"). InnerVolumeSpecName "kube-api-access-8d59h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:30:20.085733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:20.085710 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d59h\" (UniqueName: \"kubernetes.io/projected/a6125f4d-efec-4765-87e2-64d992685019-kube-api-access-8d59h\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:20.656553 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:20.656524 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" Nov 02 12:30:20.656553 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:20.656528 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl" event={"ID":"a6125f4d-efec-4765-87e2-64d992685019","Type":"ContainerDied","Data":"16b437180079ae1d111388454833fc48da322b05fb171e3f122ec72f441715c5"} Nov 02 12:30:20.656768 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:20.656597 4312 scope.go:122] "RemoveContainer" containerID="8fecfc6d27300c2063807e9c8237bcc7f855356bc7d9fe3fb407e6ee8f4a27f2" Nov 02 12:30:20.821257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:20.821227 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl"] Nov 02 12:30:20.874525 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:20.871688 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-dd087a-29893710-lv6nl"] Nov 02 12:30:21.075398 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:21.075370 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6125f4d-efec-4765-87e2-64d992685019" path="/var/lib/kubelet/pods/a6125f4d-efec-4765-87e2-64d992685019/volumes" Nov 02 12:30:21.618815 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:21.618781 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:23.165520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:23.164952 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:23.414964 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:23.414928 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:24.272210 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:24.272161 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:27.037426 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:27.037388 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:29.126911 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:29.126878 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn"] Nov 02 12:30:29.176136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:29.176107 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l"] Nov 02 12:30:29.374182 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:30:29.374144 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:30:29.390811 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:29.390744 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:31.903479 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:31.903387 4312 generic.go:356] "Generic (PLEG): container finished" podID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerID="d4fd05424c1b0ef1a11a6376b8187debdef470dda9db2d18bd2e8b8e4e358a4f" exitCode=0 Nov 02 12:30:31.903860 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:31.903476 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" event={"ID":"a65fcfca-ab6f-45bf-80b0-82a4671bc391","Type":"ContainerDied","Data":"d4fd05424c1b0ef1a11a6376b8187debdef470dda9db2d18bd2e8b8e4e358a4f"} Nov 02 12:30:31.903905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:31.903857 4312 scope.go:122] "RemoveContainer" containerID="d4fd05424c1b0ef1a11a6376b8187debdef470dda9db2d18bd2e8b8e4e358a4f" Nov 02 12:30:31.905060 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:31.905035 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:30:31.905120 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:31.905100 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:30:31.905156 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:31.905130 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:30:32.085820 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:32.085779 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:32.911001 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:32.910969 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" event={"ID":"a65fcfca-ab6f-45bf-80b0-82a4671bc391","Type":"ContainerStarted","Data":"61e37f7b1d7c7aa2c56ce1fa7f0943ff3aea625e7696e83eadf33a03a0173854"} Nov 02 12:30:33.558518 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:33.555814 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:30:33.572693 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:33.572655 4312 patch_prober.go:29] interesting pod/e2e-derive-02304760-openshift-default-5f76687f89-s72l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 12:30:33.572849 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:33.572740 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 12:30:34.557834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:34.557800 4312 patch_prober.go:29] interesting pod/e2e-derive-02304760-openshift-default-5f76687f89-s72l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 12:30:34.558407 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:34.558378 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 12:30:35.553855 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:35.553816 4312 patch_prober.go:29] interesting pod/e2e-derive-02304760-openshift-default-5f76687f89-s72l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 12:30:35.554182 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:35.554136 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 12:30:36.169387 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:36.169350 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:36.251617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:36.251588 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:30:36.556002 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:36.555905 4312 patch_prober.go:29] interesting pod/e2e-derive-02304760-openshift-default-5f76687f89-s72l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 12:30:36.556281 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:36.556238 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 12:30:37.181622 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:37.181573 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:37.555569 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:37.554927 4312 patch_prober.go:29] interesting pod/e2e-derive-02304760-openshift-default-5f76687f89-s72l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Nov 02 12:30:37.555569 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:37.555007 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Nov 02 12:30:38.683522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:38.683287 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:30:38.688701 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:38.688679 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:30:41.463691 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:41.463655 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:42.295987 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:42.295949 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:42.621835 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:42.621762 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:43.513391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:43.513345 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:43.758654 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:43.758624 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:44.009930 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.009900 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:44.146671 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.146637 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" podUID="cdc0f39f-aca8-4e76-a682-f01a1474c9f1" containerName="main" containerID="cri-o://bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b" gracePeriod=45 Nov 02 12:30:44.194852 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.194826 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" podUID="f01b2ae0-f424-46de-976f-fd2188cbfb6d" containerName="main" containerID="cri-o://e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa" gracePeriod=45 Nov 02 12:30:44.390987 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.390955 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:44.464252 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.464218 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:30:44.468064 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.468044 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:30:44.564454 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564376 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-home\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-home\") pod \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " Nov 02 12:30:44.564623 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564458 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f01b2ae0-f424-46de-976f-fd2188cbfb6d-kube-api-access-6m4dq\" (UniqueName: \"kubernetes.io/projected/f01b2ae0-f424-46de-976f-fd2188cbfb6d-kube-api-access-6m4dq\") pod \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " Nov 02 12:30:44.564623 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564517 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tmp-dir\") pod \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " Nov 02 12:30:44.564623 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564562 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tmp-dir\") pod \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " Nov 02 12:30:44.564623 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564599 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-home\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-home\") pod \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " Nov 02 12:30:44.564867 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564644 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tls-certs\") pod \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " Nov 02 12:30:44.564867 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564718 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tls-certs\") pod \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " Nov 02 12:30:44.564867 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564766 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-kube-api-access-z9tz9\" (UniqueName: \"kubernetes.io/projected/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-kube-api-access-z9tz9\") pod \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " Nov 02 12:30:44.564867 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564812 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-dshm\") pod \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " Nov 02 12:30:44.564867 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564839 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-model-cache\") pod \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\" (UID: \"cdc0f39f-aca8-4e76-a682-f01a1474c9f1\") " Nov 02 12:30:44.565125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564899 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-model-cache\") pod \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " Nov 02 12:30:44.565125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.564905 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tmp-dir" pod "cdc0f39f-aca8-4e76-a682-f01a1474c9f1" (UID: "cdc0f39f-aca8-4e76-a682-f01a1474c9f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:30:44.565617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.565264 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-home" pod "cdc0f39f-aca8-4e76-a682-f01a1474c9f1" (UID: "cdc0f39f-aca8-4e76-a682-f01a1474c9f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:30:44.565617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.565407 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.565617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.565430 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.565617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.565529 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-model-cache" pod "f01b2ae0-f424-46de-976f-fd2188cbfb6d" (UID: "f01b2ae0-f424-46de-976f-fd2188cbfb6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:30:44.565900 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.565617 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-home" pod "f01b2ae0-f424-46de-976f-fd2188cbfb6d" (UID: "f01b2ae0-f424-46de-976f-fd2188cbfb6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:30:44.565900 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.565720 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tmp-dir" pod "f01b2ae0-f424-46de-976f-fd2188cbfb6d" (UID: "f01b2ae0-f424-46de-976f-fd2188cbfb6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:30:44.565900 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.565739 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-model-cache" pod "cdc0f39f-aca8-4e76-a682-f01a1474c9f1" (UID: "cdc0f39f-aca8-4e76-a682-f01a1474c9f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:30:44.567583 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.567527 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f01b2ae0-f424-46de-976f-fd2188cbfb6d-kube-api-access-6m4dq" pod "f01b2ae0-f424-46de-976f-fd2188cbfb6d" (UID: "f01b2ae0-f424-46de-976f-fd2188cbfb6d"). InnerVolumeSpecName "kube-api-access-6m4dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:30:44.567954 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.567918 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tls-certs" pod "cdc0f39f-aca8-4e76-a682-f01a1474c9f1" (UID: "cdc0f39f-aca8-4e76-a682-f01a1474c9f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:30:44.568241 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.568215 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tls-certs" pod "f01b2ae0-f424-46de-976f-fd2188cbfb6d" (UID: "f01b2ae0-f424-46de-976f-fd2188cbfb6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:30:44.568380 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.568340 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-kube-api-access-z9tz9" pod "cdc0f39f-aca8-4e76-a682-f01a1474c9f1" (UID: "cdc0f39f-aca8-4e76-a682-f01a1474c9f1"). InnerVolumeSpecName "kube-api-access-z9tz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:30:44.568554 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.568530 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-dshm" pod "cdc0f39f-aca8-4e76-a682-f01a1474c9f1" (UID: "cdc0f39f-aca8-4e76-a682-f01a1474c9f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:30:44.665911 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.665879 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-dshm\") pod \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\" (UID: \"f01b2ae0-f424-46de-976f-fd2188cbfb6d\") " Nov 02 12:30:44.666294 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.666270 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9tz9\" (UniqueName: \"kubernetes.io/projected/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-kube-api-access-z9tz9\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.666294 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.666292 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.666597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.666302 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.666597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.666311 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.666597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.666320 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m4dq\" (UniqueName: \"kubernetes.io/projected/f01b2ae0-f424-46de-976f-fd2188cbfb6d-kube-api-access-6m4dq\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.666597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.666328 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.666597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.666336 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.666597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.666344 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc0f39f-aca8-4e76-a682-f01a1474c9f1-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.666597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.666351 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01b2ae0-f424-46de-976f-fd2188cbfb6d-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.668444 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.668412 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-dshm" pod "f01b2ae0-f424-46de-976f-fd2188cbfb6d" (UID: "f01b2ae0-f424-46de-976f-fd2188cbfb6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:30:44.767917 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.767881 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01b2ae0-f424-46de-976f-fd2188cbfb6d-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:44.985367 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.985329 4312 generic.go:356] "Generic (PLEG): container finished" podID="f01b2ae0-f424-46de-976f-fd2188cbfb6d" containerID="e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa" exitCode=0 Nov 02 12:30:44.985865 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.985589 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" event={"ID":"f01b2ae0-f424-46de-976f-fd2188cbfb6d","Type":"ContainerDied","Data":"e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa"} Nov 02 12:30:44.986059 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.986040 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" event={"ID":"f01b2ae0-f424-46de-976f-fd2188cbfb6d","Type":"ContainerDied","Data":"f5aa216b25241c32ff517ac8cd7aeec22321210094eb8f1c006dc5a07da8e6c0"} Nov 02 12:30:44.986217 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.986172 4312 scope.go:122] "RemoveContainer" containerID="e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa" Nov 02 12:30:44.986368 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.985666 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l" Nov 02 12:30:44.990203 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.990180 4312 generic.go:356] "Generic (PLEG): container finished" podID="cdc0f39f-aca8-4e76-a682-f01a1474c9f1" containerID="bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b" exitCode=0 Nov 02 12:30:44.990781 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.990757 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" event={"ID":"cdc0f39f-aca8-4e76-a682-f01a1474c9f1","Type":"ContainerDied","Data":"bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b"} Nov 02 12:30:44.991021 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.991002 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" event={"ID":"cdc0f39f-aca8-4e76-a682-f01a1474c9f1","Type":"ContainerDied","Data":"ccf9750b5c798e82025568b2b673fee7498618db8da4b0c6e8bd222ce40c71f9"} Nov 02 12:30:44.991316 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:44.991300 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn" Nov 02 12:30:45.009739 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:45.005707 4312 scope.go:122] "RemoveContainer" containerID="e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa" Nov 02 12:30:45.009739 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:30:45.006425 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa\": container with ID starting with e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa not found: ID does not exist" containerID="e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa" Nov 02 12:30:45.009739 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:45.006463 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa"} err="failed to get container status \"e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa\": rpc error: code = NotFound desc = could not find container \"e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa\": container with ID starting with e58329a8f118c35bbf26f61da619bec290e7710d8365dac976d8b0045803f4fa not found: ID does not exist" Nov 02 12:30:45.009739 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:45.006488 4312 scope.go:122] "RemoveContainer" containerID="bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b" Nov 02 12:30:45.022742 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:45.022714 4312 scope.go:122] "RemoveContainer" containerID="bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b" Nov 02 12:30:45.023204 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:30:45.023100 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b\": container with ID starting with bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b not found: ID does not exist" containerID="bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b" Nov 02 12:30:45.023204 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:45.023139 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b"} err="failed to get container status \"bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b\": rpc error: code = NotFound desc = could not find container \"bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b\": container with ID starting with bf670fbd3b1897bf8d0005e1cf6396db07e5b0ca14cd5bd9fbc0441ca7c2264b not found: ID does not exist" Nov 02 12:30:45.026746 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:45.026655 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:45.211084 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:45.211055 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn"] Nov 02 12:30:45.215154 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:45.215131 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/distinct-2-w0-b1703c-kserve-756d689cb6-jwgxn"] Nov 02 12:30:45.325391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:45.325322 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l"] Nov 02 12:30:45.336573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:45.336547 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b1703c/simulator-w0-b1703c-kserve-556bf7bff9-ddd4l"] Nov 02 12:30:45.533974 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:45.533947 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:47.075244 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:47.075209 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc0f39f-aca8-4e76-a682-f01a1474c9f1" path="/var/lib/kubelet/pods/cdc0f39f-aca8-4e76-a682-f01a1474c9f1/volumes" Nov 02 12:30:47.075953 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:47.075931 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f01b2ae0-f424-46de-976f-fd2188cbfb6d" path="/var/lib/kubelet/pods/f01b2ae0-f424-46de-976f-fd2188cbfb6d/volumes" Nov 02 12:30:47.652056 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:47.652020 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:47.899653 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:47.899626 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:48.963028 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:48.962987 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:49.956067 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:49.956019 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:50.204704 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:50.204673 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:53.052284 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:53.052239 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:53.373261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:53.373170 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:53.617076 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:53.617045 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:55.276989 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:55.276952 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:55.807081 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:55.807038 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:56.058688 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.058603 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:56.111827 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.111798 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd"] Nov 02 12:30:56.112246 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112231 4312 cpu_manager.go:414] "containerMap: removing container" podUID="cdc0f39f-aca8-4e76-a682-f01a1474c9f1" containerName="main" Nov 02 12:30:56.112297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112249 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc0f39f-aca8-4e76-a682-f01a1474c9f1" containerName="main" Nov 02 12:30:56.112297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112267 4312 cpu_manager.go:414] "containerMap: removing container" podUID="f01b2ae0-f424-46de-976f-fd2188cbfb6d" containerName="main" Nov 02 12:30:56.112297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112272 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f01b2ae0-f424-46de-976f-fd2188cbfb6d" containerName="main" Nov 02 12:30:56.112297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112280 4312 cpu_manager.go:414] "containerMap: removing container" podUID="da3d444a-4cc6-41ea-945f-8e066d221bce" containerName="revoke-keys" Nov 02 12:30:56.112297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112285 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3d444a-4cc6-41ea-945f-8e066d221bce" containerName="revoke-keys" Nov 02 12:30:56.112297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112299 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a6125f4d-efec-4765-87e2-64d992685019" containerName="cleanup" Nov 02 12:30:56.112477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112304 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6125f4d-efec-4765-87e2-64d992685019" containerName="cleanup" Nov 02 12:30:56.112477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112325 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a6125f4d-efec-4765-87e2-64d992685019" containerName="cleanup" Nov 02 12:30:56.112477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112330 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6125f4d-efec-4765-87e2-64d992685019" containerName="cleanup" Nov 02 12:30:56.112477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112398 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6125f4d-efec-4765-87e2-64d992685019" containerName="cleanup" Nov 02 12:30:56.112477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112407 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6125f4d-efec-4765-87e2-64d992685019" containerName="cleanup" Nov 02 12:30:56.112477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112414 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc0f39f-aca8-4e76-a682-f01a1474c9f1" containerName="main" Nov 02 12:30:56.112477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112421 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="f01b2ae0-f424-46de-976f-fd2188cbfb6d" containerName="main" Nov 02 12:30:56.112477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.112428 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3d444a-4cc6-41ea-945f-8e066d221bce" containerName="revoke-keys" Nov 02 12:30:56.115571 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.115555 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:30:56.167473 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.167450 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-cf0095\"" Nov 02 12:30:56.190399 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.190373 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a2a4b4-3985-4a1d-a501-d382dafeb579-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd\" (UID: \"04a2a4b4-3985-4a1d-a501-d382dafeb579\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:30:56.190595 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.190562 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7vm\" (UniqueName: \"kubernetes.io/projected/04a2a4b4-3985-4a1d-a501-d382dafeb579-kube-api-access-cg7vm\") pod \"payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd\" (UID: \"04a2a4b4-3985-4a1d-a501-d382dafeb579\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:30:56.217302 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.217272 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-cf0095-dockercfg-wgk7b\"" Nov 02 12:30:56.223409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.223384 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd"] Nov 02 12:30:56.291792 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.291756 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7vm\" (UniqueName: \"kubernetes.io/projected/04a2a4b4-3985-4a1d-a501-d382dafeb579-kube-api-access-cg7vm\") pod \"payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd\" (UID: \"04a2a4b4-3985-4a1d-a501-d382dafeb579\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:30:56.292261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.291859 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a2a4b4-3985-4a1d-a501-d382dafeb579-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd\" (UID: \"04a2a4b4-3985-4a1d-a501-d382dafeb579\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:30:56.292586 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.292565 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a2a4b4-3985-4a1d-a501-d382dafeb579-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd\" (UID: \"04a2a4b4-3985-4a1d-a501-d382dafeb579\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:30:56.356268 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.356204 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7vm\" (UniqueName: \"kubernetes.io/projected/04a2a4b4-3985-4a1d-a501-d382dafeb579-kube-api-access-cg7vm\") pod \"payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd\" (UID: \"04a2a4b4-3985-4a1d-a501-d382dafeb579\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:30:56.428147 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.428123 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:30:56.806045 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:56.805973 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd"] Nov 02 12:30:56.808673 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:30:56.808643 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a2a4b4_3985_4a1d_a501_d382dafeb579.slice/crio-d87e5efd298ad8874cdab4ae3d5113b49ccc83d2b53bf2ed77b99508a1d3c0b1 WatchSource:0}: Error finding container d87e5efd298ad8874cdab4ae3d5113b49ccc83d2b53bf2ed77b99508a1d3c0b1: Status 404 returned error can't find the container with id d87e5efd298ad8874cdab4ae3d5113b49ccc83d2b53bf2ed77b99508a1d3c0b1 Nov 02 12:30:57.060933 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.060864 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" event={"ID":"04a2a4b4-3985-4a1d-a501-d382dafeb579","Type":"ContainerStarted","Data":"f1c945a0f04a391705d065ee55839cc457031e8ab9c2c18ec8dbad9af2a4034f"} Nov 02 12:30:57.060933 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.060925 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:30:57.061077 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.060943 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" event={"ID":"04a2a4b4-3985-4a1d-a501-d382dafeb579","Type":"ContainerStarted","Data":"d87e5efd298ad8874cdab4ae3d5113b49ccc83d2b53bf2ed77b99508a1d3c0b1"} Nov 02 12:30:57.179105 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.179064 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" podStartSLOduration=1.179051122 podStartE2EDuration="1.179051122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:57.176343302 +0000 UTC m=+2074.206579551" watchObservedRunningTime="2026-11-02 12:30:57.179051122 +0000 UTC m=+2074.209287371" Nov 02 12:30:57.387417 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.387338 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942"] Nov 02 12:30:57.387840 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.387549 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" podUID="91bc9139-0856-411c-9e4e-f109316b9dbc" containerName="maas-api" containerID="cri-o://87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c" gracePeriod=30 Nov 02 12:30:57.591742 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.591708 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:57.686450 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.686429 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:30:57.811409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.811381 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91bc9139-0856-411c-9e4e-f109316b9dbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91bc9139-0856-411c-9e4e-f109316b9dbc-openshift-service-ca\") pod \"91bc9139-0856-411c-9e4e-f109316b9dbc\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " Nov 02 12:30:57.811806 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.811781 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-metrics-certs\") pod \"91bc9139-0856-411c-9e4e-f109316b9dbc\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " Nov 02 12:30:57.812125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.812102 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91bc9139-0856-411c-9e4e-f109316b9dbc-kube-api-access-ch2lt\" (UniqueName: \"kubernetes.io/projected/91bc9139-0856-411c-9e4e-f109316b9dbc-kube-api-access-ch2lt\") pod \"91bc9139-0856-411c-9e4e-f109316b9dbc\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " Nov 02 12:30:57.812406 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.812330 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-maas-api-tls\") pod \"91bc9139-0856-411c-9e4e-f109316b9dbc\" (UID: \"91bc9139-0856-411c-9e4e-f109316b9dbc\") " Nov 02 12:30:57.813872 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.813462 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91bc9139-0856-411c-9e4e-f109316b9dbc-openshift-service-ca" pod "91bc9139-0856-411c-9e4e-f109316b9dbc" (UID: "91bc9139-0856-411c-9e4e-f109316b9dbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:30:57.814251 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.814232 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91bc9139-0856-411c-9e4e-f109316b9dbc-openshift-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:57.816141 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.816114 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-metrics-certs" pod "91bc9139-0856-411c-9e4e-f109316b9dbc" (UID: "91bc9139-0856-411c-9e4e-f109316b9dbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:30:57.830543 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.822182 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-maas-api-tls" pod "91bc9139-0856-411c-9e4e-f109316b9dbc" (UID: "91bc9139-0856-411c-9e4e-f109316b9dbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:30:57.830543 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.823047 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91bc9139-0856-411c-9e4e-f109316b9dbc-kube-api-access-ch2lt" pod "91bc9139-0856-411c-9e4e-f109316b9dbc" (UID: "91bc9139-0856-411c-9e4e-f109316b9dbc"). InnerVolumeSpecName "kube-api-access-ch2lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:30:57.915393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.915312 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch2lt\" (UniqueName: \"kubernetes.io/projected/91bc9139-0856-411c-9e4e-f109316b9dbc-kube-api-access-ch2lt\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:57.915393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.915341 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-maas-api-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:57.915393 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:57.915354 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91bc9139-0856-411c-9e4e-f109316b9dbc-metrics-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:30:58.068409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:58.068371 4312 generic.go:356] "Generic (PLEG): container finished" podID="91bc9139-0856-411c-9e4e-f109316b9dbc" containerID="87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c" exitCode=0 Nov 02 12:30:58.068637 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:58.068457 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" event={"ID":"91bc9139-0856-411c-9e4e-f109316b9dbc","Type":"ContainerDied","Data":"87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c"} Nov 02 12:30:58.068637 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:58.068527 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" event={"ID":"91bc9139-0856-411c-9e4e-f109316b9dbc","Type":"ContainerDied","Data":"0d3f1c367e59a53babdd33d8c0b5f36b4d338e00cf74ebb4717b55ef6c73d686"} Nov 02 12:30:58.068637 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:58.068560 4312 scope.go:122] "RemoveContainer" containerID="87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c" Nov 02 12:30:58.068803 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:58.068758 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942" Nov 02 12:30:58.079017 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:58.078997 4312 scope.go:122] "RemoveContainer" containerID="87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c" Nov 02 12:30:58.079279 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:30:58.079263 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c\": container with ID starting with 87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c not found: ID does not exist" containerID="87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c" Nov 02 12:30:58.079333 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:58.079287 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c"} err="failed to get container status \"87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c\": rpc error: code = NotFound desc = could not find container \"87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c\": container with ID starting with 87b8cf8d41f0b18701e75d2d1d2910ae71fc6ca0bb7bc92d9a4b1e81ee7ca72c not found: ID does not exist" Nov 02 12:30:58.233383 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:58.233321 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942"] Nov 02 12:30:58.252523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:58.251431 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1703c-bcd959df-qb942"] Nov 02 12:30:59.076255 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:59.076211 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91bc9139-0856-411c-9e4e-f109316b9dbc" path="/var/lib/kubelet/pods/91bc9139-0856-411c-9e4e-f109316b9dbc/volumes" Nov 02 12:30:59.372356 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:30:59.372278 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:30:59.488664 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:59.488586 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:30:59.852456 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:30:59.852428 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:00.097048 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:00.097016 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:01.128317 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:01.128239 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:01.372423 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:01.372386 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:03.244149 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:03.244115 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-b-3a78b4_payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr_91b75ed7-022a-4d07-9d7b-a699d52d087a/payload-processing/0.log" Nov 02 12:31:04.363296 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:04.363264 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:04.647015 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:04.646931 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:04.893056 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:04.893025 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:07.377007 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:07.376974 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:07.621226 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:07.621195 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:08.072157 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:08.072130 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:31:09.248564 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:09.248521 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:09.497163 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:09.497130 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:09.642969 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:09.642876 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6"] Nov 02 12:31:09.643142 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:09.643090 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerName="istio-proxy" containerID="cri-o://3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d" gracePeriod=30 Nov 02 12:31:09.652835 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:09.652808 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp"] Nov 02 12:31:09.653014 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:09.652994 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" podUID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" containerName="istio-proxy" containerID="cri-o://a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667" gracePeriod=30 Nov 02 12:31:09.764940 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:09.764908 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:13.952310 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:13.952276 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:14.199443 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:14.199415 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:14.923161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:14.923100 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:31:14.941422 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:14.941397 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:31:15.001620 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.001594 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-credential-socket\") pod \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " Nov 02 12:31:15.001996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.001623 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-envoy\") pod \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " Nov 02 12:31:15.001996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.001642 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-token\" (UniqueName: \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-token\") pod \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " Nov 02 12:31:15.001996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.001666 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-kube-api-access-lp24x\" (UniqueName: \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-kube-api-access-lp24x\") pod \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " Nov 02 12:31:15.001996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.001700 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-certs\") pod \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " Nov 02 12:31:15.001996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.001796 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-data\") pod \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " Nov 02 12:31:15.001996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.001830 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-socket\") pod \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " Nov 02 12:31:15.001996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.001856 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-credential-socket" pod "8a53a8c4-121c-4da4-be4b-7d21a697d1f8" (UID: "8a53a8c4-121c-4da4-be4b-7d21a697d1f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:15.001996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.001931 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istiod-ca-cert\") pod \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " Nov 02 12:31:15.001996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.001997 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-podinfo\") pod \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\" (UID: \"8a53a8c4-121c-4da4-be4b-7d21a697d1f8\") " Nov 02 12:31:15.002480 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.002040 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-certs" pod "8a53a8c4-121c-4da4-be4b-7d21a697d1f8" (UID: "8a53a8c4-121c-4da4-be4b-7d21a697d1f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:15.002480 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.002124 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-socket" pod "8a53a8c4-121c-4da4-be4b-7d21a697d1f8" (UID: "8a53a8c4-121c-4da4-be4b-7d21a697d1f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:15.002480 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.002252 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-data" pod "8a53a8c4-121c-4da4-be4b-7d21a697d1f8" (UID: "8a53a8c4-121c-4da4-be4b-7d21a697d1f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:15.002663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.002490 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.002663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.002543 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.002663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.002559 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.002663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.002574 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.002895 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.002736 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istiod-ca-cert" pod "8a53a8c4-121c-4da4-be4b-7d21a697d1f8" (UID: "8a53a8c4-121c-4da4-be4b-7d21a697d1f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:31:15.003910 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.003881 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-token" pod "8a53a8c4-121c-4da4-be4b-7d21a697d1f8" (UID: "8a53a8c4-121c-4da4-be4b-7d21a697d1f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:31:15.004001 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.003935 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-envoy" pod "8a53a8c4-121c-4da4-be4b-7d21a697d1f8" (UID: "8a53a8c4-121c-4da4-be4b-7d21a697d1f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:15.004001 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.003973 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-podinfo" pod "8a53a8c4-121c-4da4-be4b-7d21a697d1f8" (UID: "8a53a8c4-121c-4da4-be4b-7d21a697d1f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:31:15.004076 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.004008 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-kube-api-access-lp24x" pod "8a53a8c4-121c-4da4-be4b-7d21a697d1f8" (UID: "8a53a8c4-121c-4da4-be4b-7d21a697d1f8"). InnerVolumeSpecName "kube-api-access-lp24x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:31:15.103427 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.103402 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-socket\") pod \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " Nov 02 12:31:15.103564 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.103439 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-certs\") pod \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " Nov 02 12:31:15.103564 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.103472 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-credential-socket\") pod \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " Nov 02 12:31:15.103564 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.103512 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-token\" (UniqueName: \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-token\") pod \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " Nov 02 12:31:15.103740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.103564 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-podinfo\") pod \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " Nov 02 12:31:15.103740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.103599 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-envoy\") pod \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " Nov 02 12:31:15.103740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.103652 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-data\") pod \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " Nov 02 12:31:15.103740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.103696 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-socket" pod "d5cc0b38-8f11-4c28-835f-c5bbe213ee19" (UID: "d5cc0b38-8f11-4c28-835f-c5bbe213ee19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:15.103740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.103707 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istiod-ca-cert\") pod \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " Nov 02 12:31:15.103994 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.103753 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-credential-socket" pod "d5cc0b38-8f11-4c28-835f-c5bbe213ee19" (UID: "d5cc0b38-8f11-4c28-835f-c5bbe213ee19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:15.103994 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.103801 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-kube-api-access-ddp5f\" (UniqueName: \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-kube-api-access-ddp5f\") pod \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\" (UID: \"d5cc0b38-8f11-4c28-835f-c5bbe213ee19\") " Nov 02 12:31:15.104103 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.104084 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-certs" pod "d5cc0b38-8f11-4c28-835f-c5bbe213ee19" (UID: "d5cc0b38-8f11-4c28-835f-c5bbe213ee19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:15.104159 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.104150 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.104213 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.104164 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.104213 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.104174 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-data" pod "d5cc0b38-8f11-4c28-835f-c5bbe213ee19" (UID: "d5cc0b38-8f11-4c28-835f-c5bbe213ee19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:15.104213 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.104174 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.104372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.104224 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.104372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.104244 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp24x\" (UniqueName: \"kubernetes.io/projected/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-kube-api-access-lp24x\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.104372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.104260 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.104372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.104274 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a53a8c4-121c-4da4-be4b-7d21a697d1f8-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.104605 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.104526 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istiod-ca-cert" pod "d5cc0b38-8f11-4c28-835f-c5bbe213ee19" (UID: "d5cc0b38-8f11-4c28-835f-c5bbe213ee19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:31:15.106099 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.106053 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-podinfo" pod "d5cc0b38-8f11-4c28-835f-c5bbe213ee19" (UID: "d5cc0b38-8f11-4c28-835f-c5bbe213ee19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:31:15.106420 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.106393 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-envoy" pod "d5cc0b38-8f11-4c28-835f-c5bbe213ee19" (UID: "d5cc0b38-8f11-4c28-835f-c5bbe213ee19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:15.106536 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.106396 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-token" pod "d5cc0b38-8f11-4c28-835f-c5bbe213ee19" (UID: "d5cc0b38-8f11-4c28-835f-c5bbe213ee19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:31:15.106611 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.106593 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-kube-api-access-ddp5f" pod "d5cc0b38-8f11-4c28-835f-c5bbe213ee19" (UID: "d5cc0b38-8f11-4c28-835f-c5bbe213ee19"). InnerVolumeSpecName "kube-api-access-ddp5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:31:15.155097 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.155072 4312 generic.go:356] "Generic (PLEG): container finished" podID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" containerID="a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667" exitCode=0 Nov 02 12:31:15.155183 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.155117 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" Nov 02 12:31:15.155183 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.155157 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" event={"ID":"d5cc0b38-8f11-4c28-835f-c5bbe213ee19","Type":"ContainerDied","Data":"a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667"} Nov 02 12:31:15.155282 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.155206 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp" event={"ID":"d5cc0b38-8f11-4c28-835f-c5bbe213ee19","Type":"ContainerDied","Data":"38f90cc0c56ee137f84f3f3c3cf007915695554dfd6b93adbf2ac6784dbbefed"} Nov 02 12:31:15.155282 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.155233 4312 scope.go:122] "RemoveContainer" containerID="a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667" Nov 02 12:31:15.156615 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.156595 4312 generic.go:356] "Generic (PLEG): container finished" podID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerID="3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d" exitCode=0 Nov 02 12:31:15.156711 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.156653 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" Nov 02 12:31:15.156711 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.156678 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" event={"ID":"8a53a8c4-121c-4da4-be4b-7d21a697d1f8","Type":"ContainerDied","Data":"3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d"} Nov 02 12:31:15.156795 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.156737 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6" event={"ID":"8a53a8c4-121c-4da4-be4b-7d21a697d1f8","Type":"ContainerDied","Data":"849d1d5581b70d92036112fc3e69b46b19d23b54122987bd5d13552e2c349544"} Nov 02 12:31:15.165480 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.165462 4312 scope.go:122] "RemoveContainer" containerID="a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667" Nov 02 12:31:15.165819 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:31:15.165802 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667\": container with ID starting with a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667 not found: ID does not exist" containerID="a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667" Nov 02 12:31:15.165911 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.165829 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667"} err="failed to get container status \"a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667\": rpc error: code = NotFound desc = could not find container \"a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667\": container with ID starting with a2f706802833246310b3fad77c1d4e0d0b4173504687bd229f76a987a6ec5667 not found: ID does not exist" Nov 02 12:31:15.165911 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.165849 4312 scope.go:122] "RemoveContainer" containerID="3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d" Nov 02 12:31:15.175056 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.175038 4312 scope.go:122] "RemoveContainer" containerID="3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d" Nov 02 12:31:15.175365 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:31:15.175345 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d\": container with ID starting with 3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d not found: ID does not exist" containerID="3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d" Nov 02 12:31:15.175433 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.175376 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d"} err="failed to get container status \"3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d\": rpc error: code = NotFound desc = could not find container \"3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d\": container with ID starting with 3ceb96ac84136c4ab303db8eab0903b0c961697ed7d02f1fc616ef1844c7eb8d not found: ID does not exist" Nov 02 12:31:15.204936 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.204916 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.205034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.204940 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.205034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.204956 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.205034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.204972 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddp5f\" (UniqueName: \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-kube-api-access-ddp5f\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.205034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.204987 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.205034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.205002 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.205034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.205017 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5cc0b38-8f11-4c28-835f-c5bbe213ee19-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:15.324032 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.324006 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6"] Nov 02 12:31:15.330144 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.330121 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-x86b6"] Nov 02 12:31:15.385752 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.385720 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:15.439323 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.439268 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp"] Nov 02 12:31:15.442894 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.442871 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1703c-openshift-default-6755bbfdc7-zdvpp"] Nov 02 12:31:15.633440 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.633404 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:15.887020 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:15.886939 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:16.041843 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:16.041806 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8"] Nov 02 12:31:16.042300 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:16.042064 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" containerID="cri-o://61e37f7b1d7c7aa2c56ce1fa7f0943ff3aea625e7696e83eadf33a03a0173854" gracePeriod=30 Nov 02 12:31:16.165808 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:16.165779 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:16.982397 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:16.982353 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:17.075096 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:17.075061 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" path="/var/lib/kubelet/pods/8a53a8c4-121c-4da4-be4b-7d21a697d1f8/volumes" Nov 02 12:31:17.075696 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:17.075676 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" path="/var/lib/kubelet/pods/d5cc0b38-8f11-4c28-835f-c5bbe213ee19/volumes" Nov 02 12:31:17.643213 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:17.643180 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:17.940694 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:17.940650 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:21.149983 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.149949 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:21.199055 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.199020 4312 generic.go:356] "Generic (PLEG): container finished" podID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerID="61e37f7b1d7c7aa2c56ce1fa7f0943ff3aea625e7696e83eadf33a03a0173854" exitCode=0 Nov 02 12:31:21.199205 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.199071 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" event={"ID":"a65fcfca-ab6f-45bf-80b0-82a4671bc391","Type":"ContainerDied","Data":"61e37f7b1d7c7aa2c56ce1fa7f0943ff3aea625e7696e83eadf33a03a0173854"} Nov 02 12:31:21.199205 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.199131 4312 scope.go:122] "RemoveContainer" containerID="d4fd05424c1b0ef1a11a6376b8187debdef470dda9db2d18bd2e8b8e4e358a4f" Nov 02 12:31:21.289965 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.289941 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:31:21.463134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.463103 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-podinfo\") pod \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " Nov 02 12:31:21.463289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.463175 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-envoy\") pod \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " Nov 02 12:31:21.463289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.463219 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-credential-socket\") pod \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " Nov 02 12:31:21.463411 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.463290 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-socket\") pod \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " Nov 02 12:31:21.463411 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.463322 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-data\") pod \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " Nov 02 12:31:21.463411 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.463399 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-kube-api-access-vk257\" (UniqueName: \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-kube-api-access-vk257\") pod \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " Nov 02 12:31:21.463579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.463451 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-token\" (UniqueName: \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-token\") pod \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " Nov 02 12:31:21.463579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.463524 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-certs\") pod \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " Nov 02 12:31:21.463579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.463553 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istiod-ca-cert\") pod \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\" (UID: \"a65fcfca-ab6f-45bf-80b0-82a4671bc391\") " Nov 02 12:31:21.464024 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.463957 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-certs" pod "a65fcfca-ab6f-45bf-80b0-82a4671bc391" (UID: "a65fcfca-ab6f-45bf-80b0-82a4671bc391"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:21.464024 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.463996 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-data" pod "a65fcfca-ab6f-45bf-80b0-82a4671bc391" (UID: "a65fcfca-ab6f-45bf-80b0-82a4671bc391"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:21.464344 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.464160 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-credential-socket" pod "a65fcfca-ab6f-45bf-80b0-82a4671bc391" (UID: "a65fcfca-ab6f-45bf-80b0-82a4671bc391"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:21.464344 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.464180 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-socket" pod "a65fcfca-ab6f-45bf-80b0-82a4671bc391" (UID: "a65fcfca-ab6f-45bf-80b0-82a4671bc391"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:21.464344 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.464259 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istiod-ca-cert" pod "a65fcfca-ab6f-45bf-80b0-82a4671bc391" (UID: "a65fcfca-ab6f-45bf-80b0-82a4671bc391"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:31:21.466020 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.465989 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-envoy" pod "a65fcfca-ab6f-45bf-80b0-82a4671bc391" (UID: "a65fcfca-ab6f-45bf-80b0-82a4671bc391"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:31:21.466258 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.466231 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-token" pod "a65fcfca-ab6f-45bf-80b0-82a4671bc391" (UID: "a65fcfca-ab6f-45bf-80b0-82a4671bc391"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:31:21.466482 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.466455 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-podinfo" pod "a65fcfca-ab6f-45bf-80b0-82a4671bc391" (UID: "a65fcfca-ab6f-45bf-80b0-82a4671bc391"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:31:21.466701 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.466681 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-kube-api-access-vk257" pod "a65fcfca-ab6f-45bf-80b0-82a4671bc391" (UID: "a65fcfca-ab6f-45bf-80b0-82a4671bc391"). InnerVolumeSpecName "kube-api-access-vk257". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:31:21.565044 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.565017 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:21.565044 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.565044 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:21.565170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.565055 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:21.565170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.565063 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:21.565170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.565071 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:21.565170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.565080 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk257\" (UniqueName: \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-kube-api-access-vk257\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:21.565170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.565092 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:21.565170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.565100 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a65fcfca-ab6f-45bf-80b0-82a4671bc391-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:21.565170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:21.565110 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a65fcfca-ab6f-45bf-80b0-82a4671bc391-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:31:22.205542 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:22.205514 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" Nov 02 12:31:22.205970 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:22.205541 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8" event={"ID":"a65fcfca-ab6f-45bf-80b0-82a4671bc391","Type":"ContainerDied","Data":"f153c2a6b51bf5bb4c3c4efdd18a5905b24330a141f84bd3d49e43fd8b542982"} Nov 02 12:31:22.205970 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:22.205600 4312 scope.go:122] "RemoveContainer" containerID="61e37f7b1d7c7aa2c56ce1fa7f0943ff3aea625e7696e83eadf33a03a0173854" Nov 02 12:31:22.418676 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:22.418651 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8"] Nov 02 12:31:22.468254 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:22.468189 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-02304760-openshift-default-5f76687f89-s72l8"] Nov 02 12:31:23.076899 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:23.076868 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" path="/var/lib/kubelet/pods/a65fcfca-ab6f-45bf-80b0-82a4671bc391/volumes" Nov 02 12:31:23.129724 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:23.129691 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:25.725713 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:25.725683 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:25.972189 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:25.972158 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:26.075142 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075066 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f"] Nov 02 12:31:26.075596 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075577 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" Nov 02 12:31:26.075704 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075597 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" Nov 02 12:31:26.075704 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075609 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a6125f4d-efec-4765-87e2-64d992685019" containerName="cleanup" Nov 02 12:31:26.075704 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075616 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6125f4d-efec-4765-87e2-64d992685019" containerName="cleanup" Nov 02 12:31:26.075704 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075647 4312 cpu_manager.go:414] "containerMap: removing container" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerName="istio-proxy" Nov 02 12:31:26.075704 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075656 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerName="istio-proxy" Nov 02 12:31:26.075704 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075670 4312 cpu_manager.go:414] "containerMap: removing container" podUID="91bc9139-0856-411c-9e4e-f109316b9dbc" containerName="maas-api" Nov 02 12:31:26.075704 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075676 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91bc9139-0856-411c-9e4e-f109316b9dbc" containerName="maas-api" Nov 02 12:31:26.075704 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075687 4312 cpu_manager.go:414] "containerMap: removing container" podUID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" containerName="istio-proxy" Nov 02 12:31:26.075704 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075694 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" containerName="istio-proxy" Nov 02 12:31:26.076136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075795 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" Nov 02 12:31:26.076136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075811 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a53a8c4-121c-4da4-be4b-7d21a697d1f8" containerName="istio-proxy" Nov 02 12:31:26.076136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075821 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="91bc9139-0856-411c-9e4e-f109316b9dbc" containerName="maas-api" Nov 02 12:31:26.076136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075832 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6125f4d-efec-4765-87e2-64d992685019" containerName="cleanup" Nov 02 12:31:26.076136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075840 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" Nov 02 12:31:26.076136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.075848 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cc0b38-8f11-4c28-835f-c5bbe213ee19" containerName="istio-proxy" Nov 02 12:31:26.080514 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.080473 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.103506 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.103471 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-maas-api-tls\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.103632 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.103616 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6519d9a2-d28f-4254-8374-c07ad6ed53e9-openshift-service-ca\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.103679 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.103644 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-metrics-certs\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.103722 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.103679 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tn6x\" (UniqueName: \"kubernetes.io/projected/6519d9a2-d28f-4254-8374-c07ad6ed53e9-kube-api-access-6tn6x\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.132487 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.132467 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-cf0095-serving-cert\"" Nov 02 12:31:26.186052 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.186006 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f"] Nov 02 12:31:26.196050 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.196020 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx"] Nov 02 12:31:26.196917 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.196871 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" Nov 02 12:31:26.197077 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.197060 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65fcfca-ab6f-45bf-80b0-82a4671bc391" containerName="istio-proxy" Nov 02 12:31:26.203987 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.203967 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:31:26.208169 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.208041 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-maas-api-tls\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.209055 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.208570 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xxsv\" (UniqueName: \"kubernetes.io/projected/b54c847f-637c-4b25-b875-b2087f210293-kube-api-access-2xxsv\") pod \"payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx\" (UID: \"b54c847f-637c-4b25-b875-b2087f210293\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:31:26.209235 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.209217 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6519d9a2-d28f-4254-8374-c07ad6ed53e9-openshift-service-ca\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.209385 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.209351 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-metrics-certs\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.209534 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.209517 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54c847f-637c-4b25-b875-b2087f210293-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx\" (UID: \"b54c847f-637c-4b25-b875-b2087f210293\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:31:26.209684 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.209664 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tn6x\" (UniqueName: \"kubernetes.io/projected/6519d9a2-d28f-4254-8374-c07ad6ed53e9-kube-api-access-6tn6x\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.209920 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:31:26.208302 4312 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-serving-cert: secret "maas-api-e2e-autores-cf0095-serving-cert" not found Nov 02 12:31:26.210008 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:31:26.209980 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-maas-api-tls podName:6519d9a2-d28f-4254-8374-c07ad6ed53e9 nodeName:}" failed. No retries permitted until 2026-11-02 12:31:26.709959133 +0000 UTC m=+2103.740195374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-maas-api-tls") pod "maas-api-e2e-autores-cf0095-74c4446f69-fft2f" (UID: "6519d9a2-d28f-4254-8374-c07ad6ed53e9") : secret "maas-api-e2e-autores-cf0095-serving-cert" not found Nov 02 12:31:26.213257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.213234 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6519d9a2-d28f-4254-8374-c07ad6ed53e9-openshift-service-ca\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.215970 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.215943 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-metrics-certs\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.243178 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.243136 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx"] Nov 02 12:31:26.256360 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.256336 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2e101a-dockercfg-npbrw\"" Nov 02 12:31:26.261346 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.261323 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2e101a\"" Nov 02 12:31:26.274276 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.274251 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tn6x\" (UniqueName: \"kubernetes.io/projected/6519d9a2-d28f-4254-8374-c07ad6ed53e9-kube-api-access-6tn6x\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.310513 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.310472 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xxsv\" (UniqueName: \"kubernetes.io/projected/b54c847f-637c-4b25-b875-b2087f210293-kube-api-access-2xxsv\") pod \"payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx\" (UID: \"b54c847f-637c-4b25-b875-b2087f210293\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:31:26.310603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.310541 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54c847f-637c-4b25-b875-b2087f210293-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx\" (UID: \"b54c847f-637c-4b25-b875-b2087f210293\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:31:26.311188 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.311169 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54c847f-637c-4b25-b875-b2087f210293-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx\" (UID: \"b54c847f-637c-4b25-b875-b2087f210293\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:31:26.372767 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.372704 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xxsv\" (UniqueName: \"kubernetes.io/projected/b54c847f-637c-4b25-b875-b2087f210293-kube-api-access-2xxsv\") pod \"payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx\" (UID: \"b54c847f-637c-4b25-b875-b2087f210293\") " pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:31:26.394018 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.393994 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:26.541269 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.541241 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:31:26.640765 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.640694 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:26.717014 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.716982 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-maas-api-tls\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.719305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.719279 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-maas-api-tls\") pod \"maas-api-e2e-autores-cf0095-74c4446f69-fft2f\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:26.920752 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.920727 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx"] Nov 02 12:31:26.922615 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:31:26.922589 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54c847f_637c_4b25_b875_b2087f210293.slice/crio-96ce837e1b80a6c86336aa958f39df31bcb4ab62a6499f3320fd641dce38da55 WatchSource:0}: Error finding container 96ce837e1b80a6c86336aa958f39df31bcb4ab62a6499f3320fd641dce38da55: Status 404 returned error can't find the container with id 96ce837e1b80a6c86336aa958f39df31bcb4ab62a6499f3320fd641dce38da55 Nov 02 12:31:26.993051 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:26.993025 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:27.238231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:27.238150 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" event={"ID":"b54c847f-637c-4b25-b875-b2087f210293","Type":"ContainerStarted","Data":"ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e"} Nov 02 12:31:27.238231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:27.238199 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" event={"ID":"b54c847f-637c-4b25-b875-b2087f210293","Type":"ContainerStarted","Data":"96ce837e1b80a6c86336aa958f39df31bcb4ab62a6499f3320fd641dce38da55"} Nov 02 12:31:27.238440 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:27.238340 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:31:27.353693 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:27.353648 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" podStartSLOduration=1.353622452 podStartE2EDuration="1.353622452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:31:27.353153323 +0000 UTC m=+2104.383389572" watchObservedRunningTime="2026-11-02 12:31:27.353622452 +0000 UTC m=+2104.383858701" Nov 02 12:31:27.405330 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:31:27.405303 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6519d9a2_d28f_4254_8374_c07ad6ed53e9.slice/crio-fc747a5d232de79151eff3509f47b1a6b96119a414afc71c59e81e71f56e7240 WatchSource:0}: Error finding container fc747a5d232de79151eff3509f47b1a6b96119a414afc71c59e81e71f56e7240: Status 404 returned error can't find the container with id fc747a5d232de79151eff3509f47b1a6b96119a414afc71c59e81e71f56e7240 Nov 02 12:31:27.406974 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:27.406946 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f"] Nov 02 12:31:27.407365 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:27.407349 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 12:31:28.246290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:28.246251 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" event={"ID":"6519d9a2-d28f-4254-8374-c07ad6ed53e9","Type":"ContainerStarted","Data":"fc747a5d232de79151eff3509f47b1a6b96119a414afc71c59e81e71f56e7240"} Nov 02 12:31:29.253318 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:29.253285 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" event={"ID":"6519d9a2-d28f-4254-8374-c07ad6ed53e9","Type":"ContainerStarted","Data":"6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68"} Nov 02 12:31:29.253741 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:29.253539 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:29.372707 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:29.372653 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" podStartSLOduration=2.517725233 podStartE2EDuration="3.372638837s" totalImagesPullingTime="854.913604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:31:29.369905992 +0000 UTC m=+2106.400142247" watchObservedRunningTime="2026-11-02 12:31:29.372638837 +0000 UTC m=+2106.402875086" Nov 02 12:31:29.373861 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:31:29.373833 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:31:31.814420 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:31.814386 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:32.626156 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:32.626119 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:35.264417 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:35.264387 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:31:37.034743 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:37.034706 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:38.250065 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:38.250034 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:31:41.936102 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:41.936063 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:46.773002 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:46.772974 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:51.968856 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:51.968822 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:56.338039 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.338000 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr"] Nov 02 12:31:56.341742 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.341726 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.393638 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.393612 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2e101a-serving-cert\"" Nov 02 12:31:56.445446 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.445414 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr"] Nov 02 12:31:56.496579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.496555 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602fd8f0-fea3-4c23-878b-0c021779fac2-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.496712 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.496618 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-metrics-certs\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.496781 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.496725 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-maas-api-tls\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.496826 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.496775 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhvn\" (UniqueName: \"kubernetes.io/projected/602fd8f0-fea3-4c23-878b-0c021779fac2-kube-api-access-qmhvn\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.597618 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.597561 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-metrics-certs\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.597618 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.597600 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-maas-api-tls\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.597771 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.597632 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhvn\" (UniqueName: \"kubernetes.io/projected/602fd8f0-fea3-4c23-878b-0c021779fac2-kube-api-access-qmhvn\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.597771 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.597719 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602fd8f0-fea3-4c23-878b-0c021779fac2-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.598321 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.598297 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602fd8f0-fea3-4c23-878b-0c021779fac2-openshift-service-ca\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.599975 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.599955 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-maas-api-tls\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.600058 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.600015 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-metrics-certs\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.634215 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.634196 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:56.658300 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.658277 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhvn\" (UniqueName: \"kubernetes.io/projected/602fd8f0-fea3-4c23-878b-0c021779fac2-kube-api-access-qmhvn\") pod \"maas-api-e2e-api-a-2e101a-575766cc75-zfhjr\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:56.876286 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.876203 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:56.954856 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:56.954832 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:57.325998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:57.325976 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr"] Nov 02 12:31:57.328161 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:31:57.328135 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602fd8f0_fea3_4c23_878b_0c021779fac2.slice/crio-6d50ffab1744d319a66e916ce80b27ac793d8c9c47e9541761f68b5258f3dfa2 WatchSource:0}: Error finding container 6d50ffab1744d319a66e916ce80b27ac793d8c9c47e9541761f68b5258f3dfa2: Status 404 returned error can't find the container with id 6d50ffab1744d319a66e916ce80b27ac793d8c9c47e9541761f68b5258f3dfa2 Nov 02 12:31:57.394041 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:57.394014 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" event={"ID":"602fd8f0-fea3-4c23-878b-0c021779fac2","Type":"ContainerStarted","Data":"6d50ffab1744d319a66e916ce80b27ac793d8c9c47e9541761f68b5258f3dfa2"} Nov 02 12:31:58.401012 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:58.400943 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" event={"ID":"602fd8f0-fea3-4c23-878b-0c021779fac2","Type":"ContainerStarted","Data":"75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598"} Nov 02 12:31:58.401353 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:58.401042 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:31:58.521261 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:58.521209 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" podStartSLOduration=1.735157346 podStartE2EDuration="2.521193257s" totalImagesPullingTime="786.035911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:31:58.517428817 +0000 UTC m=+2135.547665067" watchObservedRunningTime="2026-11-02 12:31:58.521193257 +0000 UTC m=+2135.551429508" Nov 02 12:31:58.878428 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:58.878397 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:59.128544 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:31:59.128443 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:31:59.373135 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:31:59.373094 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:32:01.690086 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:01.690056 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:04.416750 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:04.416724 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:32:06.587171 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:06.587137 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:11.628192 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:11.628158 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:15.746186 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:15.746156 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:16.489672 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:16.489636 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:16.852103 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:16.852027 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:19.290220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.290190 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j"] Nov 02 12:32:19.293760 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.293743 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.393402 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.393374 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-cf0095\"/\"openshift-service-ca.crt\"" Nov 02 12:32:19.393569 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.393454 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-cf0095\"/\"default-dockercfg-kp79h\"" Nov 02 12:32:19.395269 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.395252 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-cf0095\"/\"autores-model-cf0095-kserve-self-signed-certs\"" Nov 02 12:32:19.395269 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.395251 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-cf0095\"/\"kube-root-ca.crt\"" Nov 02 12:32:19.397573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.397554 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j"] Nov 02 12:32:19.406754 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.406732 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tls-certs\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.406837 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.406770 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-model-cache\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.406885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.406864 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv2qv\" (UniqueName: \"kubernetes.io/projected/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-kube-api-access-lv2qv\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.406967 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.406952 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-home\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.407012 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.407001 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-dshm\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.407075 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.407059 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tmp-dir\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.507909 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.507874 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-home\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.508076 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.507942 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-dshm\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.508076 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.507979 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tmp-dir\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.508076 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.508030 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tls-certs\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.508076 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.508072 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-model-cache\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.508288 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.508126 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv2qv\" (UniqueName: \"kubernetes.io/projected/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-kube-api-access-lv2qv\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.508352 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.508336 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-home\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.508449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.508419 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tmp-dir\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.508566 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.508544 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-model-cache\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.510817 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.510796 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-dshm\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.511192 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.511167 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tls-certs\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.541444 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.541366 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:19.569779 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.569748 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv2qv\" (UniqueName: \"kubernetes.io/projected/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-kube-api-access-lv2qv\") pod \"autores-model-cf0095-kserve-5567969f65-jtg8j\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.606310 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.606284 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:19.993156 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:19.993132 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j"] Nov 02 12:32:19.995287 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:32:19.995256 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0933577_3dbb_4e46_94a6_f3fcb5fb4dd5.slice/crio-57f0954b408369998ff206cfca77bcdbc6add527ecf4c2062011be21cd5e68f9 WatchSource:0}: Error finding container 57f0954b408369998ff206cfca77bcdbc6add527ecf4c2062011be21cd5e68f9: Status 404 returned error can't find the container with id 57f0954b408369998ff206cfca77bcdbc6add527ecf4c2062011be21cd5e68f9 Nov 02 12:32:20.521744 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:20.521712 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" event={"ID":"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5","Type":"ContainerStarted","Data":"01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0"} Nov 02 12:32:20.522120 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:20.521753 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" event={"ID":"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5","Type":"ContainerStarted","Data":"57f0954b408369998ff206cfca77bcdbc6add527ecf4c2062011be21cd5e68f9"} Nov 02 12:32:20.606708 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:20.606686 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:20.639212 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:20.639162 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" podStartSLOduration=1.639149162 podStartE2EDuration="1.639149162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:20.637626753 +0000 UTC m=+2157.667863003" watchObservedRunningTime="2026-11-02 12:32:20.639149162 +0000 UTC m=+2157.669385412" Nov 02 12:32:21.803041 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:21.802984 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:26.694766 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:26.694718 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:29.372924 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:32:29.372891 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:32:29.606701 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:29.606671 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:29.619751 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:29.619724 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:30.587461 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:30.587435 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:32:31.616227 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:31.616198 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:36.422908 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:36.422849 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:41.381483 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:41.381433 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:43.918371 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:43.918336 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:44.620370 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:44.620338 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:45.225801 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:45.225770 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:45.622704 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:45.622601 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:46.575694 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:46.575667 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:50.050605 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:50.050570 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:50.307094 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:50.307021 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:51.666161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:51.666128 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:53.233303 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:53.233274 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:53.485263 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:53.485168 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:56.429620 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:56.429575 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:56.797050 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:32:56.796962 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:32:59.371251 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:32:59.371219 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:33:01.290796 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:01.290765 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg"] Nov 02 12:33:01.342125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:01.341128 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k"] Nov 02 12:33:01.394077 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:01.394043 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j"] Nov 02 12:33:01.532145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:01.532111 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:01.778305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:01.778251 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:02.022033 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:02.022005 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:03.628557 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:03.628521 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:06.588600 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:06.588491 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:07.005130 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:07.005093 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:10.149153 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:10.149083 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:10.735802 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:10.735771 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:10.985427 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:10.985398 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:11.818847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:11.818817 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:16.085172 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.085142 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:16.310369 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.310330 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" containerID="cri-o://bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295" gracePeriod=45 Nov 02 12:33:16.366702 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.366618 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" containerID="cri-o://b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b" gracePeriod=45 Nov 02 12:33:16.401047 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.401023 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:16.415648 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.415620 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" podUID="b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" containerName="main" containerID="cri-o://01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0" gracePeriod=45 Nov 02 12:33:16.577284 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.577237 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" podUID="b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": dial tcp 10.134.0.89:8000: connect: connection refused" Nov 02 12:33:16.621135 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.621079 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:33:16.700765 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.700737 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:33:16.707463 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.707440 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:33:16.747289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747238 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-dshm\") pod \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " Nov 02 12:33:16.747459 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747291 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tls-certs\") pod \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " Nov 02 12:33:16.747459 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747322 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-home\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-home\") pod \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " Nov 02 12:33:16.747459 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747351 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tmp-dir\") pod \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " Nov 02 12:33:16.747459 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747420 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-kube-api-access-lv2qv\" (UniqueName: \"kubernetes.io/projected/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-kube-api-access-lv2qv\") pod \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747469 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-home\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-home\") pod \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747514 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tmp-dir\") pod \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747585 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-model-cache\") pod \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747618 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-model-cache\") pod \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747651 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-dshm\") pod \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747681 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-kube-api-access-m44q9\" (UniqueName: \"kubernetes.io/projected/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-kube-api-access-m44q9\") pod \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747731 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tls-certs\") pod \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\" (UID: \"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747752 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tls-certs\") pod \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747789 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-model-cache\") pod \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747809 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-kube-api-access-lv8wp\" (UniqueName: \"kubernetes.io/projected/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-kube-api-access-lv8wp\") pod \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747847 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-dshm\") pod \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747865 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tmp-dir\") pod \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\" (UID: \"ec93f568-75f8-4ff3-8718-1c3e86fe5b49\") " Nov 02 12:33:16.747941 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.747895 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-home\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-home\") pod \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\" (UID: \"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab\") " Nov 02 12:33:16.748535 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.748352 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-home" pod "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" (UID: "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.748787 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.748763 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-model-cache" pod "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" (UID: "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.748991 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.748965 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-home" pod "ec93f568-75f8-4ff3-8718-1c3e86fe5b49" (UID: "ec93f568-75f8-4ff3-8718-1c3e86fe5b49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.749232 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.749204 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-model-cache" pod "ec93f568-75f8-4ff3-8718-1c3e86fe5b49" (UID: "ec93f568-75f8-4ff3-8718-1c3e86fe5b49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.750469 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.749610 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-model-cache" pod "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" (UID: "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.750469 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.749746 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-home" pod "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" (UID: "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.750469 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.749793 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tmp-dir" pod "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" (UID: "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.750469 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.749945 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tmp-dir" pod "ec93f568-75f8-4ff3-8718-1c3e86fe5b49" (UID: "ec93f568-75f8-4ff3-8718-1c3e86fe5b49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.750973 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.750951 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tmp-dir" pod "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" (UID: "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.751109 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.750971 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-dshm" pod "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" (UID: "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.751443 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.751406 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-dshm" pod "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" (UID: "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.752527 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.752472 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tls-certs" pod "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" (UID: "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:33:16.752660 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.752551 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-kube-api-access-lv2qv" pod "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" (UID: "b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5"). InnerVolumeSpecName "kube-api-access-lv2qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:33:16.753109 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.753087 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tls-certs" pod "ec93f568-75f8-4ff3-8718-1c3e86fe5b49" (UID: "ec93f568-75f8-4ff3-8718-1c3e86fe5b49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:33:16.753406 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.753365 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-dshm" pod "ec93f568-75f8-4ff3-8718-1c3e86fe5b49" (UID: "ec93f568-75f8-4ff3-8718-1c3e86fe5b49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:16.753834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.753811 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-kube-api-access-lv8wp" pod "ec93f568-75f8-4ff3-8718-1c3e86fe5b49" (UID: "ec93f568-75f8-4ff3-8718-1c3e86fe5b49"). InnerVolumeSpecName "kube-api-access-lv8wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:33:16.754192 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.754166 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tls-certs" pod "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" (UID: "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:33:16.754663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.754641 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-kube-api-access-m44q9" pod "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" (UID: "7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab"). InnerVolumeSpecName "kube-api-access-m44q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:33:16.826965 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.826934 4312 generic.go:356] "Generic (PLEG): container finished" podID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerID="bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295" exitCode=0 Nov 02 12:33:16.827130 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.826973 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" Nov 02 12:33:16.827130 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.827011 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" event={"ID":"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab","Type":"ContainerDied","Data":"bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295"} Nov 02 12:33:16.827130 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.827052 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg" event={"ID":"7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab","Type":"ContainerDied","Data":"4fd99b5a1b52f552a45ed25f6b04bf4c66520a736bee86c3a8a351cf89738010"} Nov 02 12:33:16.827130 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.827067 4312 scope.go:122] "RemoveContainer" containerID="bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295" Nov 02 12:33:16.828421 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.828398 4312 generic.go:356] "Generic (PLEG): container finished" podID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerID="b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b" exitCode=0 Nov 02 12:33:16.828549 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.828448 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" Nov 02 12:33:16.828549 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.828477 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" event={"ID":"ec93f568-75f8-4ff3-8718-1c3e86fe5b49","Type":"ContainerDied","Data":"b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b"} Nov 02 12:33:16.828549 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.828542 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" event={"ID":"ec93f568-75f8-4ff3-8718-1c3e86fe5b49","Type":"ContainerDied","Data":"d9503d6feb998ea90b8f9f68477b30dcc109df0a7829bb93c8e014d3b1bbfbca"} Nov 02 12:33:16.829966 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.829937 4312 generic.go:356] "Generic (PLEG): container finished" podID="b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" containerID="01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0" exitCode=0 Nov 02 12:33:16.830068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.829974 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" Nov 02 12:33:16.830068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.830022 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" event={"ID":"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5","Type":"ContainerDied","Data":"01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0"} Nov 02 12:33:16.830068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.830053 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j" event={"ID":"b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5","Type":"ContainerDied","Data":"57f0954b408369998ff206cfca77bcdbc6add527ecf4c2062011be21cd5e68f9"} Nov 02 12:33:16.836381 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.836365 4312 scope.go:122] "RemoveContainer" containerID="bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295" Nov 02 12:33:16.836657 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:33:16.836637 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295\": container with ID starting with bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295 not found: ID does not exist" containerID="bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295" Nov 02 12:33:16.836739 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.836666 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295"} err="failed to get container status \"bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295\": rpc error: code = NotFound desc = could not find container \"bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295\": container with ID starting with bea6d3551806d486283087821afb5deca0b002d4952f3960362db0bc6de20295 not found: ID does not exist" Nov 02 12:33:16.836739 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.836683 4312 scope.go:122] "RemoveContainer" containerID="b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b" Nov 02 12:33:16.845769 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.845752 4312 scope.go:122] "RemoveContainer" containerID="b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b" Nov 02 12:33:16.846019 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:33:16.846001 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b\": container with ID starting with b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b not found: ID does not exist" containerID="b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b" Nov 02 12:33:16.846055 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.846027 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b"} err="failed to get container status \"b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b\": rpc error: code = NotFound desc = could not find container \"b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b\": container with ID starting with b715735fe900f547b9c067d6ce4044cf8d7fe94ea269e5d19cf52c738302a67b not found: ID does not exist" Nov 02 12:33:16.846055 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.846044 4312 scope.go:122] "RemoveContainer" containerID="01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0" Nov 02 12:33:16.849392 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849371 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849394 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849407 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849421 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849434 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv2qv\" (UniqueName: \"kubernetes.io/projected/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-kube-api-access-lv2qv\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849455 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849468 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849480 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849522 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849536 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849549 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m44q9\" (UniqueName: \"kubernetes.io/projected/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-kube-api-access-m44q9\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849563 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849576 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849590 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849604 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv8wp\" (UniqueName: \"kubernetes.io/projected/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-kube-api-access-lv8wp\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849618 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849631 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec93f568-75f8-4ff3-8718-1c3e86fe5b49-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.849800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.849643 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:16.853604 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.853473 4312 scope.go:122] "RemoveContainer" containerID="01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0" Nov 02 12:33:16.853935 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:33:16.853903 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0\": container with ID starting with 01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0 not found: ID does not exist" containerID="01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0" Nov 02 12:33:16.854053 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.853938 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0"} err="failed to get container status \"01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0\": rpc error: code = NotFound desc = could not find container \"01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0\": container with ID starting with 01a772b6106cf07ef55437b31da45903d9b662185caf72e21ed7ff97b5f01eb0 not found: ID does not exist" Nov 02 12:33:16.895711 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:16.895637 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:17.087622 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:17.087570 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j"] Nov 02 12:33:17.091068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:17.091045 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-cf0095/autores-model-cf0095-kserve-5567969f65-jtg8j"] Nov 02 12:33:17.197206 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:17.197180 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k"] Nov 02 12:33:17.200690 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:17.200668 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k"] Nov 02 12:33:17.308557 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:17.308533 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg"] Nov 02 12:33:17.312825 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:17.312805 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4b9210/distinct-w3-4b9210-kserve-6c849bffdb-jvwtg"] Nov 02 12:33:17.644283 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:17.644197 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4b9210/premium-w3-4b9210-kserve-59dcbb6fcf-pjk7k" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Nov 02 12:33:19.074361 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:19.074330 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" path="/var/lib/kubelet/pods/7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab/volumes" Nov 02 12:33:19.074793 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:19.074777 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" path="/var/lib/kubelet/pods/b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5/volumes" Nov 02 12:33:19.075136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:19.075125 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" path="/var/lib/kubelet/pods/ec93f568-75f8-4ff3-8718-1c3e86fe5b49/volumes" Nov 02 12:33:21.698834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:21.698798 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:22.438437 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:22.438404 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:23.153566 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:23.153526 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684"] Nov 02 12:33:23.154043 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:23.153779 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerName="istio-proxy" containerID="cri-o://821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9" gracePeriod=30 Nov 02 12:33:26.557435 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:26.557403 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:26.804304 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:26.804266 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:27.050028 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:27.049993 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:27.440575 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:27.440542 4312 patch_prober.go:29] interesting pod/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 12:33:27.440739 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:27.440593 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 12:33:28.428855 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.428801 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:33:28.560957 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.560932 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-socket\") pod \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " Nov 02 12:33:28.561097 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.560970 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-podinfo\") pod \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " Nov 02 12:33:28.561097 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.560997 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-credential-socket\") pod \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " Nov 02 12:33:28.561097 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561030 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-certs\") pod \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " Nov 02 12:33:28.561225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561113 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istiod-ca-cert\") pod \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " Nov 02 12:33:28.561225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561150 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-kube-api-access-2b6sc\" (UniqueName: \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-kube-api-access-2b6sc\") pod \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " Nov 02 12:33:28.561325 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561223 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-socket" pod "2a481cdc-c7c2-417e-aebd-582ed8c6cee2" (UID: "2a481cdc-c7c2-417e-aebd-582ed8c6cee2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:28.561325 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561249 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-envoy\") pod \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " Nov 02 12:33:28.561325 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561282 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-data\") pod \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " Nov 02 12:33:28.561489 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561334 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-token\" (UniqueName: \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-token\") pod \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\" (UID: \"2a481cdc-c7c2-417e-aebd-582ed8c6cee2\") " Nov 02 12:33:28.561489 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561354 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-credential-socket" pod "2a481cdc-c7c2-417e-aebd-582ed8c6cee2" (UID: "2a481cdc-c7c2-417e-aebd-582ed8c6cee2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:28.561489 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561400 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-certs" pod "2a481cdc-c7c2-417e-aebd-582ed8c6cee2" (UID: "2a481cdc-c7c2-417e-aebd-582ed8c6cee2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:28.561489 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561454 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istiod-ca-cert" pod "2a481cdc-c7c2-417e-aebd-582ed8c6cee2" (UID: "2a481cdc-c7c2-417e-aebd-582ed8c6cee2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:33:28.561794 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561692 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-data" pod "2a481cdc-c7c2-417e-aebd-582ed8c6cee2" (UID: "2a481cdc-c7c2-417e-aebd-582ed8c6cee2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:28.561794 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561773 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:28.561794 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561788 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:28.561947 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561798 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:28.561947 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.561810 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:28.563437 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.563406 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-token" pod "2a481cdc-c7c2-417e-aebd-582ed8c6cee2" (UID: "2a481cdc-c7c2-417e-aebd-582ed8c6cee2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:33:28.563437 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.563422 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-envoy" pod "2a481cdc-c7c2-417e-aebd-582ed8c6cee2" (UID: "2a481cdc-c7c2-417e-aebd-582ed8c6cee2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:33:28.563656 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.563638 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-kube-api-access-2b6sc" pod "2a481cdc-c7c2-417e-aebd-582ed8c6cee2" (UID: "2a481cdc-c7c2-417e-aebd-582ed8c6cee2"). InnerVolumeSpecName "kube-api-access-2b6sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:33:28.563723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.563701 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-podinfo" pod "2a481cdc-c7c2-417e-aebd-582ed8c6cee2" (UID: "2a481cdc-c7c2-417e-aebd-582ed8c6cee2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:33:28.663227 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.663201 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:28.663227 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.663226 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:28.663388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.663236 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b6sc\" (UniqueName: \"kubernetes.io/projected/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-kube-api-access-2b6sc\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:28.663388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.663247 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:28.663388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.663255 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a481cdc-c7c2-417e-aebd-582ed8c6cee2-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:28.894723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.894627 4312 generic.go:356] "Generic (PLEG): container finished" podID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerID="821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9" exitCode=0 Nov 02 12:33:28.894723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.894685 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" Nov 02 12:33:28.894903 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.894729 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" event={"ID":"2a481cdc-c7c2-417e-aebd-582ed8c6cee2","Type":"ContainerDied","Data":"821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9"} Nov 02 12:33:28.894903 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.894830 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684" event={"ID":"2a481cdc-c7c2-417e-aebd-582ed8c6cee2","Type":"ContainerDied","Data":"10545560d69273e2174b704a1ac9efb2d5746fca6199591ce2ee821bc484f8c2"} Nov 02 12:33:28.894903 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.894856 4312 scope.go:122] "RemoveContainer" containerID="821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9" Nov 02 12:33:28.905099 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.905081 4312 scope.go:122] "RemoveContainer" containerID="821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9" Nov 02 12:33:28.905462 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:33:28.905435 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9\": container with ID starting with 821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9 not found: ID does not exist" containerID="821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9" Nov 02 12:33:28.905581 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:28.905475 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9"} err="failed to get container status \"821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9\": rpc error: code = NotFound desc = could not find container \"821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9\": container with ID starting with 821d97be8b1fd75893080658c4a77f5e5b206fc2be042f97755161d8c211c7a9 not found: ID does not exist" Nov 02 12:33:29.063010 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:29.062981 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684"] Nov 02 12:33:29.112461 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:29.112433 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-3a78b4-openshift-default-77b8648d45-2d684"] Nov 02 12:33:29.375010 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:33:29.374978 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:33:31.075367 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:31.075339 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" path="/var/lib/kubelet/pods/2a481cdc-c7c2-417e-aebd-582ed8c6cee2/volumes" Nov 02 12:33:31.686043 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:31.686011 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:36.618298 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.618261 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:36.763238 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763210 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl"] Nov 02 12:33:36.763679 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763666 4312 cpu_manager.go:414] "containerMap: removing container" podUID="b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" containerName="main" Nov 02 12:33:36.763727 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763680 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" containerName="main" Nov 02 12:33:36.763727 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763696 4312 cpu_manager.go:414] "containerMap: removing container" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" Nov 02 12:33:36.763727 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763701 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" Nov 02 12:33:36.763727 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763709 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" Nov 02 12:33:36.763727 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763713 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" Nov 02 12:33:36.763877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763734 4312 cpu_manager.go:414] "containerMap: removing container" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerName="istio-proxy" Nov 02 12:33:36.763877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763739 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerName="istio-proxy" Nov 02 12:33:36.763877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763812 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a481cdc-c7c2-417e-aebd-582ed8c6cee2" containerName="istio-proxy" Nov 02 12:33:36.763877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763821 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0933577-3dbb-4e46-94a6-f3fcb5fb4dd5" containerName="main" Nov 02 12:33:36.763877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763828 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db5c59e-e7cc-4bb5-97c8-dfb5fc8a99ab" containerName="main" Nov 02 12:33:36.763877 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.763833 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec93f568-75f8-4ff3-8718-1c3e86fe5b49" containerName="main" Nov 02 12:33:36.768195 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.768180 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:36.871948 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.871880 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl"] Nov 02 12:33:36.874784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.874764 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4b9210-openshift-default-dockercfg-fqcg6\"" Nov 02 12:33:36.939076 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.939043 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:36.939231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.939085 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:36.939231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.939114 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:36.939231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.939203 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:36.939354 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.939233 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh7zr\" (UniqueName: \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-kube-api-access-fh7zr\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:36.939409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.939387 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:36.939446 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.939424 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:36.939536 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.939518 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:36.939611 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:36.939596 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040222 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040194 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040345 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040234 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040345 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040254 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040345 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040289 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040345 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040312 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh7zr\" (UniqueName: \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-kube-api-access-fh7zr\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040585 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040375 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040585 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040399 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040585 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040425 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040585 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040477 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040776 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040648 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040776 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040692 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.040951 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.040925 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.041136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.041119 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.041179 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.041157 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.042644 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.042619 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.042801 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.042783 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.101075 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.101052 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh7zr\" (UniqueName: \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-kube-api-access-fh7zr\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.101075 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.101060 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.381188 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.381162 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:37.806697 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.806669 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl"] Nov 02 12:33:37.808887 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:33:37.808864 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe06105_7e54_47b5_acf5_a5181ea5cd9e.slice/crio-75fa1ff2b305639e263e796b5c87a8eb8f876922128647f6862c0330a51bffa3 WatchSource:0}: Error finding container 75fa1ff2b305639e263e796b5c87a8eb8f876922128647f6862c0330a51bffa3: Status 404 returned error can't find the container with id 75fa1ff2b305639e263e796b5c87a8eb8f876922128647f6862c0330a51bffa3 Nov 02 12:33:37.811110 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.811079 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:33:37.811170 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.811148 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:33:37.811215 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.811178 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:33:37.943833 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.943796 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" event={"ID":"6fe06105-7e54-47b5-acf5-a5181ea5cd9e","Type":"ContainerStarted","Data":"d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937"} Nov 02 12:33:37.944009 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:37.943857 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" event={"ID":"6fe06105-7e54-47b5-acf5-a5181ea5cd9e","Type":"ContainerStarted","Data":"75fa1ff2b305639e263e796b5c87a8eb8f876922128647f6862c0330a51bffa3"} Nov 02 12:33:38.065105 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:38.065018 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" podStartSLOduration=2.065004987 podStartE2EDuration="2.065004987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:33:38.062809594 +0000 UTC m=+2235.093045846" watchObservedRunningTime="2026-11-02 12:33:38.065004987 +0000 UTC m=+2235.095241237" Nov 02 12:33:38.381622 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:38.381543 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:38.383095 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:38.383070 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 12:33:38.383230 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:38.383115 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 12:33:38.422452 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:38.422411 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:38.674080 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:38.674040 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:39.382377 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:39.382341 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 12:33:39.382800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:39.382396 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 12:33:39.539852 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:39.539817 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:40.381953 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:40.381915 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 12:33:40.382146 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:40.381980 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 12:33:41.381939 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:41.381903 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 12:33:41.382345 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:41.381956 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 12:33:41.501958 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:41.501914 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:42.382535 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:42.382482 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 12:33:42.382982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:42.382560 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 12:33:43.382065 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:43.382032 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 12:33:43.382240 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:43.382082 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 12:33:44.233793 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:44.233762 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:44.382428 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:44.382392 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 12:33:44.382589 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:44.382456 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 12:33:44.766965 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:44.766928 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd"] Nov 02 12:33:45.130843 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:45.130760 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:45.382871 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:45.382775 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:45.659630 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:45.659542 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:45.821603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:45.821575 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:45.821927 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:45.821907 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:45.823288 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:45.823267 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:33:45.910206 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:45.910152 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:46.540750 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:46.540716 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:49.767350 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:49.767311 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" podUID="04a2a4b4-3985-4a1d-a501-d382dafeb579" containerName="payload-pre-processing" containerID="cri-o://f1c945a0f04a391705d065ee55839cc457031e8ab9c2c18ec8dbad9af2a4034f" gracePeriod=25 Nov 02 12:33:50.026653 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.026591 4312 generic.go:356] "Generic (PLEG): container finished" podID="04a2a4b4-3985-4a1d-a501-d382dafeb579" containerID="f1c945a0f04a391705d065ee55839cc457031e8ab9c2c18ec8dbad9af2a4034f" exitCode=0 Nov 02 12:33:50.026766 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.026650 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" event={"ID":"04a2a4b4-3985-4a1d-a501-d382dafeb579","Type":"ContainerDied","Data":"f1c945a0f04a391705d065ee55839cc457031e8ab9c2c18ec8dbad9af2a4034f"} Nov 02 12:33:50.026766 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.026689 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" event={"ID":"04a2a4b4-3985-4a1d-a501-d382dafeb579","Type":"ContainerDied","Data":"d87e5efd298ad8874cdab4ae3d5113b49ccc83d2b53bf2ed77b99508a1d3c0b1"} Nov 02 12:33:50.026766 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.026706 4312 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d87e5efd298ad8874cdab4ae3d5113b49ccc83d2b53bf2ed77b99508a1d3c0b1" Nov 02 12:33:50.030685 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.030667 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:33:50.073017 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.072989 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a2a4b4-3985-4a1d-a501-d382dafeb579-kube-api-access-cg7vm\" (UniqueName: \"kubernetes.io/projected/04a2a4b4-3985-4a1d-a501-d382dafeb579-kube-api-access-cg7vm\") pod \"04a2a4b4-3985-4a1d-a501-d382dafeb579\" (UID: \"04a2a4b4-3985-4a1d-a501-d382dafeb579\") " Nov 02 12:33:50.073147 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.073056 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a2a4b4-3985-4a1d-a501-d382dafeb579-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a2a4b4-3985-4a1d-a501-d382dafeb579-plugins-config-volume\") pod \"04a2a4b4-3985-4a1d-a501-d382dafeb579\" (UID: \"04a2a4b4-3985-4a1d-a501-d382dafeb579\") " Nov 02 12:33:50.073405 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.073383 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a2a4b4-3985-4a1d-a501-d382dafeb579-plugins-config-volume" pod "04a2a4b4-3985-4a1d-a501-d382dafeb579" (UID: "04a2a4b4-3985-4a1d-a501-d382dafeb579"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:33:50.075106 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.075088 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a2a4b4-3985-4a1d-a501-d382dafeb579-kube-api-access-cg7vm" pod "04a2a4b4-3985-4a1d-a501-d382dafeb579" (UID: "04a2a4b4-3985-4a1d-a501-d382dafeb579"). InnerVolumeSpecName "kube-api-access-cg7vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:33:50.086357 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.086335 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:50.174929 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.174817 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg7vm\" (UniqueName: \"kubernetes.io/projected/04a2a4b4-3985-4a1d-a501-d382dafeb579-kube-api-access-cg7vm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:50.174929 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.174849 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04a2a4b4-3985-4a1d-a501-d382dafeb579-plugins-config-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:50.834946 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.834909 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf"] Nov 02 12:33:50.835425 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:50.835103 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" podUID="ef82d8ca-0100-4d9f-a449-ada84c603c33" containerName="maas-api" containerID="cri-o://61ebbb71b32c9fc611b4ab9770a5792b70fdf9810394260795d84d47dbd5b820" gracePeriod=30 Nov 02 12:33:51.034530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.034467 4312 generic.go:356] "Generic (PLEG): container finished" podID="ef82d8ca-0100-4d9f-a449-ada84c603c33" containerID="61ebbb71b32c9fc611b4ab9770a5792b70fdf9810394260795d84d47dbd5b820" exitCode=0 Nov 02 12:33:51.034695 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.034557 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" event={"ID":"ef82d8ca-0100-4d9f-a449-ada84c603c33","Type":"ContainerDied","Data":"61ebbb71b32c9fc611b4ab9770a5792b70fdf9810394260795d84d47dbd5b820"} Nov 02 12:33:51.034695 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.034614 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd" Nov 02 12:33:51.081803 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.081727 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:51.112442 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.112419 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:33:51.184970 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.184946 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef82d8ca-0100-4d9f-a449-ada84c603c33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef82d8ca-0100-4d9f-a449-ada84c603c33-openshift-service-ca\") pod \"ef82d8ca-0100-4d9f-a449-ada84c603c33\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " Nov 02 12:33:51.185093 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.185056 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-maas-api-tls\") pod \"ef82d8ca-0100-4d9f-a449-ada84c603c33\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " Nov 02 12:33:51.185093 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.185085 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-metrics-certs\") pod \"ef82d8ca-0100-4d9f-a449-ada84c603c33\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " Nov 02 12:33:51.185207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.185107 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef82d8ca-0100-4d9f-a449-ada84c603c33-kube-api-access-sgrhm\" (UniqueName: \"kubernetes.io/projected/ef82d8ca-0100-4d9f-a449-ada84c603c33-kube-api-access-sgrhm\") pod \"ef82d8ca-0100-4d9f-a449-ada84c603c33\" (UID: \"ef82d8ca-0100-4d9f-a449-ada84c603c33\") " Nov 02 12:33:51.185290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.185269 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef82d8ca-0100-4d9f-a449-ada84c603c33-openshift-service-ca" pod "ef82d8ca-0100-4d9f-a449-ada84c603c33" (UID: "ef82d8ca-0100-4d9f-a449-ada84c603c33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:33:51.185379 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.185365 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef82d8ca-0100-4d9f-a449-ada84c603c33-openshift-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:51.187250 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.187225 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef82d8ca-0100-4d9f-a449-ada84c603c33-kube-api-access-sgrhm" pod "ef82d8ca-0100-4d9f-a449-ada84c603c33" (UID: "ef82d8ca-0100-4d9f-a449-ada84c603c33"). InnerVolumeSpecName "kube-api-access-sgrhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:33:51.187250 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.187238 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-maas-api-tls" pod "ef82d8ca-0100-4d9f-a449-ada84c603c33" (UID: "ef82d8ca-0100-4d9f-a449-ada84c603c33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:33:51.187409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.187225 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-metrics-certs" pod "ef82d8ca-0100-4d9f-a449-ada84c603c33" (UID: "ef82d8ca-0100-4d9f-a449-ada84c603c33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:33:51.198782 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.198760 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd"] Nov 02 12:33:51.204386 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.204362 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-cf0095-795f99687f-f5gwd"] Nov 02 12:33:51.286388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.286357 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-maas-api-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:51.286388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.286382 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef82d8ca-0100-4d9f-a449-ada84c603c33-metrics-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:51.286388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.286393 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgrhm\" (UniqueName: \"kubernetes.io/projected/ef82d8ca-0100-4d9f-a449-ada84c603c33-kube-api-access-sgrhm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:33:51.842758 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.842670 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf"] Nov 02 12:33:51.843130 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.843093 4312 cpu_manager.go:414] "containerMap: removing container" podUID="04a2a4b4-3985-4a1d-a501-d382dafeb579" containerName="payload-pre-processing" Nov 02 12:33:51.843130 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.843104 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a2a4b4-3985-4a1d-a501-d382dafeb579" containerName="payload-pre-processing" Nov 02 12:33:51.843220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.843140 4312 cpu_manager.go:414] "containerMap: removing container" podUID="ef82d8ca-0100-4d9f-a449-ada84c603c33" containerName="maas-api" Nov 02 12:33:51.843220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.843146 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef82d8ca-0100-4d9f-a449-ada84c603c33" containerName="maas-api" Nov 02 12:33:51.843220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.843219 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef82d8ca-0100-4d9f-a449-ada84c603c33" containerName="maas-api" Nov 02 12:33:51.843317 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.843227 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a2a4b4-3985-4a1d-a501-d382dafeb579" containerName="payload-pre-processing" Nov 02 12:33:51.846375 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.846353 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.893798 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.893763 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.893928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.893815 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.893928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.893877 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.894013 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.893994 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.894048 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.894036 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.894165 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.894147 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.894227 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.894212 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9835cb-d925-45f4-b33a-413735e2ef60-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.894263 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.894253 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9835cb-d925-45f4-b33a-413735e2ef60-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.894377 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.894359 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkhv\" (UniqueName: \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-kube-api-access-8nkhv\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.950510 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.950479 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf"] Nov 02 12:33:51.995095 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995071 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995095 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995104 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995123 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995143 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995283 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995371 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995412 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9835cb-d925-45f4-b33a-413735e2ef60-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995450 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9835cb-d925-45f4-b33a-413735e2ef60-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995637 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995531 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkhv\" (UniqueName: \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-kube-api-access-8nkhv\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995637 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995569 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-data\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995637 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995607 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995773 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995660 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-credential-socket\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.995773 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.995713 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-certs\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.996065 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.996041 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9835cb-d925-45f4-b33a-413735e2ef60-istiod-ca-cert\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.997897 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.997868 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9835cb-d925-45f4-b33a-413735e2ef60-istio-podinfo\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:51.997979 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:51.997913 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-envoy\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:52.041247 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.041224 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" event={"ID":"ef82d8ca-0100-4d9f-a449-ada84c603c33","Type":"ContainerDied","Data":"2afb115e55591d2b1a66e7288521b7ce7834bc25757466fe919d915d676056a8"} Nov 02 12:33:52.041247 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.041240 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf" Nov 02 12:33:52.041397 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.041276 4312 scope.go:122] "RemoveContainer" containerID="61ebbb71b32c9fc611b4ab9770a5792b70fdf9810394260795d84d47dbd5b820" Nov 02 12:33:52.055616 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.055589 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-istio-token\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:52.055834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.055814 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkhv\" (UniqueName: \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-kube-api-access-8nkhv\") pod \"e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:52.160677 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.160630 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:52.206012 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.205970 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf"] Nov 02 12:33:52.208143 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.208124 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-4b9210-75969765db-5n7xf"] Nov 02 12:33:52.586911 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.586890 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf"] Nov 02 12:33:52.588395 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:33:52.588361 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9835cb_d925_45f4_b33a_413735e2ef60.slice/crio-2ef3c8757ed0f2a730e62f3636f87979a348b32e0daeb7bd261c46ba4134eec9 WatchSource:0}: Error finding container 2ef3c8757ed0f2a730e62f3636f87979a348b32e0daeb7bd261c46ba4134eec9: Status 404 returned error can't find the container with id 2ef3c8757ed0f2a730e62f3636f87979a348b32e0daeb7bd261c46ba4134eec9 Nov 02 12:33:52.590843 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.590813 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:33:52.590901 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.590879 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:33:52.590936 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.590910 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 12:33:52.893353 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:52.893259 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:53.050390 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:53.049544 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" event={"ID":"af9835cb-d925-45f4-b33a-413735e2ef60","Type":"ContainerStarted","Data":"d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce"} Nov 02 12:33:53.050390 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:53.049652 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" event={"ID":"af9835cb-d925-45f4-b33a-413735e2ef60","Type":"ContainerStarted","Data":"2ef3c8757ed0f2a730e62f3636f87979a348b32e0daeb7bd261c46ba4134eec9"} Nov 02 12:33:53.084189 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:53.084154 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a2a4b4-3985-4a1d-a501-d382dafeb579" path="/var/lib/kubelet/pods/04a2a4b4-3985-4a1d-a501-d382dafeb579/volumes" Nov 02 12:33:53.084610 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:53.084596 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef82d8ca-0100-4d9f-a449-ada84c603c33" path="/var/lib/kubelet/pods/ef82d8ca-0100-4d9f-a449-ada84c603c33/volumes" Nov 02 12:33:53.167578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:53.164887 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:53.173901 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:53.173864 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 12:33:53.174091 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:53.173921 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" podUID="af9835cb-d925-45f4-b33a-413735e2ef60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 12:33:53.174751 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:53.174706 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" podStartSLOduration=2.1746925 podStartE2EDuration="2.1746925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:33:53.173048674 +0000 UTC m=+2250.203284925" watchObservedRunningTime="2026-11-02 12:33:53.1746925 +0000 UTC m=+2250.204928750" Nov 02 12:33:54.167896 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:54.167859 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 12:33:54.168489 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:54.168455 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" podUID="af9835cb-d925-45f4-b33a-413735e2ef60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 12:33:55.172523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:55.171585 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 12:33:55.172523 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:55.171649 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" podUID="af9835cb-d925-45f4-b33a-413735e2ef60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 12:33:56.172522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:56.168874 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 12:33:56.172522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:56.168935 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" podUID="af9835cb-d925-45f4-b33a-413735e2ef60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 12:33:57.165677 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:57.162703 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:57.170519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:57.166465 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 12:33:57.170519 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:57.166541 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" podUID="af9835cb-d925-45f4-b33a-413735e2ef60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 12:33:57.412421 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:57.412384 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:57.980766 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:57.980719 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:58.161896 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:58.161862 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Nov 02 12:33:58.162549 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:58.162489 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" podUID="af9835cb-d925-45f4-b33a-413735e2ef60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Nov 02 12:33:58.731368 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:58.731335 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:59.165708 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:59.165683 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:59.165996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:59.165975 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:59.166082 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:59.166065 4312 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" secret="" err="secret \"e2e-worker-w3-4b9210-openshift-default-dockercfg-fqcg6\" not found" Nov 02 12:33:59.166796 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:59.166775 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:33:59.167530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:59.167481 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl"] Nov 02 12:33:59.167705 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:59.167685 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" containerID="cri-o://d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937" gracePeriod=30 Nov 02 12:33:59.181549 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:59.181522 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf"] Nov 02 12:33:59.261264 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:33:59.261236 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:33:59.375919 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:33:59.375874 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:34:00.108149 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:00.108105 4312 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" secret="" err="secret \"e2e-worker-w3-4b9210-openshift-default-dockercfg-fqcg6\" not found" Nov 02 12:34:00.824568 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:00.824539 4312 patch_prober.go:29] interesting pod/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 12:34:00.824734 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:00.824586 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 12:34:01.112623 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:01.112540 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" podUID="af9835cb-d925-45f4-b33a-413735e2ef60" containerName="istio-proxy" containerID="cri-o://d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce" gracePeriod=30 Nov 02 12:34:02.254663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.254623 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f"] Nov 02 12:34:02.255146 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.254877 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" podUID="6519d9a2-d28f-4254-8374-c07ad6ed53e9" containerName="maas-api" containerID="cri-o://6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68" gracePeriod=30 Nov 02 12:34:02.479300 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.479061 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:02.514713 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.514661 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:34:02.612158 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.612133 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-metrics-certs\") pod \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " Nov 02 12:34:02.612269 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.612197 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6519d9a2-d28f-4254-8374-c07ad6ed53e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6519d9a2-d28f-4254-8374-c07ad6ed53e9-openshift-service-ca\") pod \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " Nov 02 12:34:02.612269 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.612258 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6519d9a2-d28f-4254-8374-c07ad6ed53e9-kube-api-access-6tn6x\" (UniqueName: \"kubernetes.io/projected/6519d9a2-d28f-4254-8374-c07ad6ed53e9-kube-api-access-6tn6x\") pod \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " Nov 02 12:34:02.612375 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.612360 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-maas-api-tls\") pod \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\" (UID: \"6519d9a2-d28f-4254-8374-c07ad6ed53e9\") " Nov 02 12:34:02.612623 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.612595 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6519d9a2-d28f-4254-8374-c07ad6ed53e9-openshift-service-ca" pod "6519d9a2-d28f-4254-8374-c07ad6ed53e9" (UID: "6519d9a2-d28f-4254-8374-c07ad6ed53e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:34:02.612748 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.612698 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6519d9a2-d28f-4254-8374-c07ad6ed53e9-openshift-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:02.614453 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.614428 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-metrics-certs" pod "6519d9a2-d28f-4254-8374-c07ad6ed53e9" (UID: "6519d9a2-d28f-4254-8374-c07ad6ed53e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:34:02.614586 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.614455 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-maas-api-tls" pod "6519d9a2-d28f-4254-8374-c07ad6ed53e9" (UID: "6519d9a2-d28f-4254-8374-c07ad6ed53e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:34:02.614586 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.614460 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6519d9a2-d28f-4254-8374-c07ad6ed53e9-kube-api-access-6tn6x" pod "6519d9a2-d28f-4254-8374-c07ad6ed53e9" (UID: "6519d9a2-d28f-4254-8374-c07ad6ed53e9"). InnerVolumeSpecName "kube-api-access-6tn6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:34:02.713934 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.713902 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-maas-api-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:02.713934 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.713929 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6519d9a2-d28f-4254-8374-c07ad6ed53e9-metrics-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:02.713934 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:02.713939 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tn6x\" (UniqueName: \"kubernetes.io/projected/6519d9a2-d28f-4254-8374-c07ad6ed53e9-kube-api-access-6tn6x\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:03.123664 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:03.123636 4312 generic.go:356] "Generic (PLEG): container finished" podID="6519d9a2-d28f-4254-8374-c07ad6ed53e9" containerID="6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68" exitCode=0 Nov 02 12:34:03.123664 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:03.123667 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" Nov 02 12:34:03.123846 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:03.123710 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" event={"ID":"6519d9a2-d28f-4254-8374-c07ad6ed53e9","Type":"ContainerDied","Data":"6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68"} Nov 02 12:34:03.123846 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:03.123761 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f" event={"ID":"6519d9a2-d28f-4254-8374-c07ad6ed53e9","Type":"ContainerDied","Data":"fc747a5d232de79151eff3509f47b1a6b96119a414afc71c59e81e71f56e7240"} Nov 02 12:34:03.123846 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:03.123785 4312 scope.go:122] "RemoveContainer" containerID="6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68" Nov 02 12:34:03.132627 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:03.132606 4312 scope.go:122] "RemoveContainer" containerID="6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68" Nov 02 12:34:03.132905 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:03.132888 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68\": container with ID starting with 6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68 not found: ID does not exist" containerID="6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68" Nov 02 12:34:03.132947 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:03.132914 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68"} err="failed to get container status \"6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68\": rpc error: code = NotFound desc = could not find container \"6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68\": container with ID starting with 6a39c735c3b8fc646580dfe07ded8beca7c24a31152484e8b04e6d87c6821d68 not found: ID does not exist" Nov 02 12:34:03.287284 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:03.287252 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f"] Nov 02 12:34:03.336884 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:03.336815 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-cf0095-74c4446f69-fft2f"] Nov 02 12:34:04.341454 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.341415 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:04.417083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.417059 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:34:04.534032 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.533958 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-token\" (UniqueName: \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-token\") pod \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " Nov 02 12:34:04.534032 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534012 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-credential-socket\") pod \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " Nov 02 12:34:04.534228 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534074 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-data\") pod \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " Nov 02 12:34:04.534288 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534233 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-podinfo\") pod \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " Nov 02 12:34:04.534288 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534278 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-kube-api-access-fh7zr\" (UniqueName: \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-kube-api-access-fh7zr\") pod \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " Nov 02 12:34:04.534391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534279 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-credential-socket" pod "6fe06105-7e54-47b5-acf5-a5181ea5cd9e" (UID: "6fe06105-7e54-47b5-acf5-a5181ea5cd9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:34:04.534391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534316 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-data" pod "6fe06105-7e54-47b5-acf5-a5181ea5cd9e" (UID: "6fe06105-7e54-47b5-acf5-a5181ea5cd9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:34:04.534391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534320 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-socket\") pod \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " Nov 02 12:34:04.534391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534374 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-envoy\") pod \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " Nov 02 12:34:04.534634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534398 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-certs\") pod \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " Nov 02 12:34:04.534634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534418 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istiod-ca-cert\") pod \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\" (UID: \"6fe06105-7e54-47b5-acf5-a5181ea5cd9e\") " Nov 02 12:34:04.534634 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534570 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-socket" pod "6fe06105-7e54-47b5-acf5-a5181ea5cd9e" (UID: "6fe06105-7e54-47b5-acf5-a5181ea5cd9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:34:04.534870 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534853 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:04.534954 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534876 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:04.534954 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534890 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:04.534954 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534913 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-certs" pod "6fe06105-7e54-47b5-acf5-a5181ea5cd9e" (UID: "6fe06105-7e54-47b5-acf5-a5181ea5cd9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:34:04.534954 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.534919 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istiod-ca-cert" pod "6fe06105-7e54-47b5-acf5-a5181ea5cd9e" (UID: "6fe06105-7e54-47b5-acf5-a5181ea5cd9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:34:04.536257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.536232 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-token" pod "6fe06105-7e54-47b5-acf5-a5181ea5cd9e" (UID: "6fe06105-7e54-47b5-acf5-a5181ea5cd9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:34:04.536432 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.536411 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-podinfo" pod "6fe06105-7e54-47b5-acf5-a5181ea5cd9e" (UID: "6fe06105-7e54-47b5-acf5-a5181ea5cd9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:34:04.536554 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.536535 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-kube-api-access-fh7zr" pod "6fe06105-7e54-47b5-acf5-a5181ea5cd9e" (UID: "6fe06105-7e54-47b5-acf5-a5181ea5cd9e"). InnerVolumeSpecName "kube-api-access-fh7zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:34:04.536678 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.536662 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-envoy" pod "6fe06105-7e54-47b5-acf5-a5181ea5cd9e" (UID: "6fe06105-7e54-47b5-acf5-a5181ea5cd9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:34:04.635992 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.635961 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:04.635992 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.635989 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh7zr\" (UniqueName: \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-kube-api-access-fh7zr\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:04.636195 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.636003 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:04.636195 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.636012 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:04.636195 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.636021 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:04.636195 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:04.636029 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fe06105-7e54-47b5-acf5-a5181ea5cd9e-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:05.075394 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.075351 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6519d9a2-d28f-4254-8374-c07ad6ed53e9" path="/var/lib/kubelet/pods/6519d9a2-d28f-4254-8374-c07ad6ed53e9/volumes" Nov 02 12:34:05.135969 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.135944 4312 generic.go:356] "Generic (PLEG): container finished" podID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerID="d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937" exitCode=0 Nov 02 12:34:05.136068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.135985 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" Nov 02 12:34:05.136068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.136008 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" event={"ID":"6fe06105-7e54-47b5-acf5-a5181ea5cd9e","Type":"ContainerDied","Data":"d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937"} Nov 02 12:34:05.136068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.136065 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl" event={"ID":"6fe06105-7e54-47b5-acf5-a5181ea5cd9e","Type":"ContainerDied","Data":"75fa1ff2b305639e263e796b5c87a8eb8f876922128647f6862c0330a51bffa3"} Nov 02 12:34:05.136186 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.136091 4312 scope.go:122] "RemoveContainer" containerID="d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937" Nov 02 12:34:05.144891 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.144876 4312 scope.go:122] "RemoveContainer" containerID="d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937" Nov 02 12:34:05.145128 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:05.145113 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937\": container with ID starting with d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937 not found: ID does not exist" containerID="d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937" Nov 02 12:34:05.145176 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.145137 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937"} err="failed to get container status \"d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937\": rpc error: code = NotFound desc = could not find container \"d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937\": container with ID starting with d5e362ec5a55b888f980fc243823b1165439d890e7e6a5b34bf33b0042934937 not found: ID does not exist" Nov 02 12:34:05.361245 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.361169 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl"] Nov 02 12:34:05.410556 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.410532 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-qd4kl"] Nov 02 12:34:05.475572 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.475539 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:05.722928 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:05.722901 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:06.365168 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.365145 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:34:06.450781 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.450751 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9835cb-d925-45f4-b33a-413735e2ef60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9835cb-d925-45f4-b33a-413735e2ef60-istiod-ca-cert\") pod \"af9835cb-d925-45f4-b33a-413735e2ef60\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " Nov 02 12:34:06.450902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.450807 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-istio-token\" (UniqueName: \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-istio-token\") pod \"af9835cb-d925-45f4-b33a-413735e2ef60\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " Nov 02 12:34:06.450902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.450846 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-certs\") pod \"af9835cb-d925-45f4-b33a-413735e2ef60\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " Nov 02 12:34:06.450902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.450886 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-socket\") pod \"af9835cb-d925-45f4-b33a-413735e2ef60\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " Nov 02 12:34:06.451060 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.450922 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-data\") pod \"af9835cb-d925-45f4-b33a-413735e2ef60\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " Nov 02 12:34:06.451060 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.450948 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-credential-socket\") pod \"af9835cb-d925-45f4-b33a-413735e2ef60\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " Nov 02 12:34:06.451060 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.450991 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-kube-api-access-8nkhv\" (UniqueName: \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-kube-api-access-8nkhv\") pod \"af9835cb-d925-45f4-b33a-413735e2ef60\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " Nov 02 12:34:06.451060 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.451055 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-envoy\") pod \"af9835cb-d925-45f4-b33a-413735e2ef60\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " Nov 02 12:34:06.451204 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.451085 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af9835cb-d925-45f4-b33a-413735e2ef60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9835cb-d925-45f4-b33a-413735e2ef60-istio-podinfo\") pod \"af9835cb-d925-45f4-b33a-413735e2ef60\" (UID: \"af9835cb-d925-45f4-b33a-413735e2ef60\") " Nov 02 12:34:06.451204 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.451149 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9835cb-d925-45f4-b33a-413735e2ef60-istiod-ca-cert" pod "af9835cb-d925-45f4-b33a-413735e2ef60" (UID: "af9835cb-d925-45f4-b33a-413735e2ef60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:34:06.451401 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.451385 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9835cb-d925-45f4-b33a-413735e2ef60-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:06.453748 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.453673 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-certs" pod "af9835cb-d925-45f4-b33a-413735e2ef60" (UID: "af9835cb-d925-45f4-b33a-413735e2ef60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:34:06.453865 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.453772 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-socket" pod "af9835cb-d925-45f4-b33a-413735e2ef60" (UID: "af9835cb-d925-45f4-b33a-413735e2ef60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:34:06.453865 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.453793 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-data" pod "af9835cb-d925-45f4-b33a-413735e2ef60" (UID: "af9835cb-d925-45f4-b33a-413735e2ef60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:34:06.453865 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.453851 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-credential-socket" pod "af9835cb-d925-45f4-b33a-413735e2ef60" (UID: "af9835cb-d925-45f4-b33a-413735e2ef60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:34:06.455671 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.455642 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-istio-token" pod "af9835cb-d925-45f4-b33a-413735e2ef60" (UID: "af9835cb-d925-45f4-b33a-413735e2ef60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:34:06.455772 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.455649 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-envoy" pod "af9835cb-d925-45f4-b33a-413735e2ef60" (UID: "af9835cb-d925-45f4-b33a-413735e2ef60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:34:06.455772 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.455742 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af9835cb-d925-45f4-b33a-413735e2ef60-istio-podinfo" pod "af9835cb-d925-45f4-b33a-413735e2ef60" (UID: "af9835cb-d925-45f4-b33a-413735e2ef60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:34:06.455852 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.455814 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-kube-api-access-8nkhv" pod "af9835cb-d925-45f4-b33a-413735e2ef60" (UID: "af9835cb-d925-45f4-b33a-413735e2ef60"). InnerVolumeSpecName "kube-api-access-8nkhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:34:06.552230 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.552206 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:06.552230 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.552228 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:06.552416 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.552237 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:06.552416 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.552246 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:06.552416 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.552255 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nkhv\" (UniqueName: \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-kube-api-access-8nkhv\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:06.552416 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.552265 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9835cb-d925-45f4-b33a-413735e2ef60-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:06.552416 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.552273 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9835cb-d925-45f4-b33a-413735e2ef60-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:06.552416 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:06.552282 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9835cb-d925-45f4-b33a-413735e2ef60-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:07.075768 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.075737 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" path="/var/lib/kubelet/pods/6fe06105-7e54-47b5-acf5-a5181ea5cd9e/volumes" Nov 02 12:34:07.148793 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.148763 4312 generic.go:356] "Generic (PLEG): container finished" podID="af9835cb-d925-45f4-b33a-413735e2ef60" containerID="d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce" exitCode=0 Nov 02 12:34:07.148936 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.148813 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" Nov 02 12:34:07.148936 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.148851 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" event={"ID":"af9835cb-d925-45f4-b33a-413735e2ef60","Type":"ContainerDied","Data":"d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce"} Nov 02 12:34:07.148936 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.148908 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf" event={"ID":"af9835cb-d925-45f4-b33a-413735e2ef60","Type":"ContainerDied","Data":"2ef3c8757ed0f2a730e62f3636f87979a348b32e0daeb7bd261c46ba4134eec9"} Nov 02 12:34:07.149085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.148937 4312 scope.go:122] "RemoveContainer" containerID="d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce" Nov 02 12:34:07.154299 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.154274 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:07.157881 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.157858 4312 scope.go:122] "RemoveContainer" containerID="d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce" Nov 02 12:34:07.158166 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:07.158148 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce\": container with ID starting with d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce not found: ID does not exist" containerID="d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce" Nov 02 12:34:07.158234 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.158174 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce"} err="failed to get container status \"d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce\": rpc error: code = NotFound desc = could not find container \"d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce\": container with ID starting with d1f8dd19547570387ed132524caad385b047893a9b633c85c1c5f3fc347950ce not found: ID does not exist" Nov 02 12:34:07.363256 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.363180 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf"] Nov 02 12:34:07.412007 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.411979 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4b9210-openshift-default-dcf4c64c-t2ltf"] Nov 02 12:34:07.677224 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:07.677186 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:08.310530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:08.310483 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:09.075367 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:09.075327 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9835cb-d925-45f4-b33a-413735e2ef60" path="/var/lib/kubelet/pods/af9835cb-d925-45f4-b33a-413735e2ef60/volumes" Nov 02 12:34:09.232803 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:09.232767 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:09.662540 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:09.662510 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:09.920284 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:09.920203 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:11.500595 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:11.500563 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:12.531146 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:12.531117 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-b-3a78b4_payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr_91b75ed7-022a-4d07-9d7b-a699d52d087a/payload-processing/0.log" Nov 02 12:34:12.954894 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:12.954864 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:13.621513 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:13.621467 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:13.949607 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:13.949573 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:14.949217 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:14.949182 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:15.793484 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:15.793452 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:16.427290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:16.427256 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:16.912469 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:16.912400 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:17.155251 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:17.155171 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:18.426064 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:18.426018 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:18.673632 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:18.673601 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:19.621280 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:19.621249 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:20.086279 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:20.086243 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:21.566750 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.566710 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg"] Nov 02 12:34:21.567418 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.567391 4312 cpu_manager.go:414] "containerMap: removing container" podUID="af9835cb-d925-45f4-b33a-413735e2ef60" containerName="istio-proxy" Nov 02 12:34:21.567532 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.567420 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9835cb-d925-45f4-b33a-413735e2ef60" containerName="istio-proxy" Nov 02 12:34:21.567532 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.567469 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" Nov 02 12:34:21.567532 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.567478 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" Nov 02 12:34:21.567532 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.567516 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6519d9a2-d28f-4254-8374-c07ad6ed53e9" containerName="maas-api" Nov 02 12:34:21.567532 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.567527 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6519d9a2-d28f-4254-8374-c07ad6ed53e9" containerName="maas-api" Nov 02 12:34:21.567787 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.567639 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe06105-7e54-47b5-acf5-a5181ea5cd9e" containerName="istio-proxy" Nov 02 12:34:21.567787 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.567652 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6519d9a2-d28f-4254-8374-c07ad6ed53e9" containerName="maas-api" Nov 02 12:34:21.567787 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.567666 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9835cb-d925-45f4-b33a-413735e2ef60" containerName="istio-proxy" Nov 02 12:34:21.570976 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.570949 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.676648 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.676595 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg"] Nov 02 12:34:21.688159 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.688130 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.688282 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.688187 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.688348 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.688276 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.688348 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.688332 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rd4\" (UniqueName: \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-kube-api-access-k4rd4\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.789017 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.788985 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rd4\" (UniqueName: \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-kube-api-access-k4rd4\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.789141 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.789078 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.789207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.789139 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.789207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.789173 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.789811 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.789793 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.789890 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.789838 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.791510 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.791476 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.849873 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.849820 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rd4\" (UniqueName: \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-kube-api-access-k4rd4\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:21.884464 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:21.884446 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:22.261982 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:22.261962 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg"] Nov 02 12:34:22.263826 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:34:22.263794 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a368c1_41d7_41bf_a4b7_a36e38e0958d.slice/crio-a6d564e8fd4bdf4167fd8634ffbcb4d89a2e97f8e35715528cf97b7c19692f7b WatchSource:0}: Error finding container a6d564e8fd4bdf4167fd8634ffbcb4d89a2e97f8e35715528cf97b7c19692f7b: Status 404 returned error can't find the container with id a6d564e8fd4bdf4167fd8634ffbcb4d89a2e97f8e35715528cf97b7c19692f7b Nov 02 12:34:22.458979 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:22.458949 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:22.974894 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:22.974865 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx"] Nov 02 12:34:23.234885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:23.234806 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" event={"ID":"10a368c1-41d7-41bf-a4b7-a36e38e0958d","Type":"ContainerStarted","Data":"5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575"} Nov 02 12:34:23.234885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:23.234860 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:23.234885 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:23.234876 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" event={"ID":"10a368c1-41d7-41bf-a4b7-a36e38e0958d","Type":"ContainerStarted","Data":"a6d564e8fd4bdf4167fd8634ffbcb4d89a2e97f8e35715528cf97b7c19692f7b"} Nov 02 12:34:23.351728 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:23.351676 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" podStartSLOduration=2.351661249 podStartE2EDuration="2.351661249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:23.350635798 +0000 UTC m=+2280.380872049" watchObservedRunningTime="2026-11-02 12:34:23.351661249 +0000 UTC m=+2280.381897520" Nov 02 12:34:23.447621 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:23.447592 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:23.498884 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:23.498818 4312 scope.go:122] "RemoveContainer" containerID="4d415d30d13a1b416da0feb9da6cf1639c2238856f139bf7bf212bd30e776d69" Nov 02 12:34:23.695794 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:23.695768 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:25.636427 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:25.636392 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:25.879172 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:25.879133 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:27.975644 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:27.975606 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" podUID="b54c847f-637c-4b25-b875-b2087f210293" containerName="payload-processing" containerID="cri-o://ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e" gracePeriod=25 Nov 02 12:34:28.225903 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.225850 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:34:28.258006 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.257981 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54c847f-637c-4b25-b875-b2087f210293-kube-api-access-2xxsv\" (UniqueName: \"kubernetes.io/projected/b54c847f-637c-4b25-b875-b2087f210293-kube-api-access-2xxsv\") pod \"b54c847f-637c-4b25-b875-b2087f210293\" (UID: \"b54c847f-637c-4b25-b875-b2087f210293\") " Nov 02 12:34:28.258143 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.258042 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b54c847f-637c-4b25-b875-b2087f210293-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54c847f-637c-4b25-b875-b2087f210293-plugins-config-volume\") pod \"b54c847f-637c-4b25-b875-b2087f210293\" (UID: \"b54c847f-637c-4b25-b875-b2087f210293\") " Nov 02 12:34:28.258409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.258388 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54c847f-637c-4b25-b875-b2087f210293-plugins-config-volume" pod "b54c847f-637c-4b25-b875-b2087f210293" (UID: "b54c847f-637c-4b25-b875-b2087f210293"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:34:28.260189 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.260167 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54c847f-637c-4b25-b875-b2087f210293-kube-api-access-2xxsv" pod "b54c847f-637c-4b25-b875-b2087f210293" (UID: "b54c847f-637c-4b25-b875-b2087f210293"). InnerVolumeSpecName "kube-api-access-2xxsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:34:28.260300 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.260206 4312 generic.go:356] "Generic (PLEG): container finished" podID="b54c847f-637c-4b25-b875-b2087f210293" containerID="ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e" exitCode=0 Nov 02 12:34:28.260345 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.260311 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" Nov 02 12:34:28.260386 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.260339 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" event={"ID":"b54c847f-637c-4b25-b875-b2087f210293","Type":"ContainerDied","Data":"ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e"} Nov 02 12:34:28.260420 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.260392 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx" event={"ID":"b54c847f-637c-4b25-b875-b2087f210293","Type":"ContainerDied","Data":"96ce837e1b80a6c86336aa958f39df31bcb4ab62a6499f3320fd641dce38da55"} Nov 02 12:34:28.260454 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.260423 4312 scope.go:122] "RemoveContainer" containerID="ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e" Nov 02 12:34:28.273725 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.273705 4312 scope.go:122] "RemoveContainer" containerID="ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e" Nov 02 12:34:28.273962 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:28.273944 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e\": container with ID starting with ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e not found: ID does not exist" containerID="ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e" Nov 02 12:34:28.274008 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.273971 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e"} err="failed to get container status \"ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e\": rpc error: code = NotFound desc = could not find container \"ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e\": container with ID starting with ee6f472ce8ef4a13d726631ed4cc6b2a6f4edeb75510422cd6d6e7dbfd153a1e not found: ID does not exist" Nov 02 12:34:28.359223 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.359179 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xxsv\" (UniqueName: \"kubernetes.io/projected/b54c847f-637c-4b25-b875-b2087f210293-kube-api-access-2xxsv\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:28.359223 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.359216 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54c847f-637c-4b25-b875-b2087f210293-plugins-config-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:28.451304 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.451275 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx"] Nov 02 12:34:28.455428 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.455407 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2e101a-5457bf9f75-9nhkx"] Nov 02 12:34:28.758626 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:28.758593 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:29.077113 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.077022 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b54c847f-637c-4b25-b875-b2087f210293" path="/var/lib/kubelet/pods/b54c847f-637c-4b25-b875-b2087f210293/volumes" Nov 02 12:34:29.373299 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:29.373218 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:34:29.624529 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.624418 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr"] Nov 02 12:34:29.624740 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.624713 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" podUID="602fd8f0-fea3-4c23-878b-0c021779fac2" containerName="maas-api" containerID="cri-o://75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598" gracePeriod=30 Nov 02 12:34:29.831937 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.831777 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:29.893282 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.893257 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:34:29.972656 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.972628 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602fd8f0-fea3-4c23-878b-0c021779fac2-kube-api-access-qmhvn\" (UniqueName: \"kubernetes.io/projected/602fd8f0-fea3-4c23-878b-0c021779fac2-kube-api-access-qmhvn\") pod \"602fd8f0-fea3-4c23-878b-0c021779fac2\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " Nov 02 12:34:29.972792 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.972700 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-metrics-certs\") pod \"602fd8f0-fea3-4c23-878b-0c021779fac2\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " Nov 02 12:34:29.972792 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.972743 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-maas-api-tls\") pod \"602fd8f0-fea3-4c23-878b-0c021779fac2\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " Nov 02 12:34:29.972792 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.972773 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602fd8f0-fea3-4c23-878b-0c021779fac2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602fd8f0-fea3-4c23-878b-0c021779fac2-openshift-service-ca\") pod \"602fd8f0-fea3-4c23-878b-0c021779fac2\" (UID: \"602fd8f0-fea3-4c23-878b-0c021779fac2\") " Nov 02 12:34:29.973168 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.973139 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602fd8f0-fea3-4c23-878b-0c021779fac2-openshift-service-ca" pod "602fd8f0-fea3-4c23-878b-0c021779fac2" (UID: "602fd8f0-fea3-4c23-878b-0c021779fac2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:34:29.974707 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.974681 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602fd8f0-fea3-4c23-878b-0c021779fac2-kube-api-access-qmhvn" pod "602fd8f0-fea3-4c23-878b-0c021779fac2" (UID: "602fd8f0-fea3-4c23-878b-0c021779fac2"). InnerVolumeSpecName "kube-api-access-qmhvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:34:29.974800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.974723 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-metrics-certs" pod "602fd8f0-fea3-4c23-878b-0c021779fac2" (UID: "602fd8f0-fea3-4c23-878b-0c021779fac2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:34:29.974800 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:29.974765 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-maas-api-tls" pod "602fd8f0-fea3-4c23-878b-0c021779fac2" (UID: "602fd8f0-fea3-4c23-878b-0c021779fac2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:34:30.073857 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.073829 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-metrics-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:30.073857 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.073853 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602fd8f0-fea3-4c23-878b-0c021779fac2-maas-api-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:30.074018 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.073864 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602fd8f0-fea3-4c23-878b-0c021779fac2-openshift-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:30.074018 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.073872 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmhvn\" (UniqueName: \"kubernetes.io/projected/602fd8f0-fea3-4c23-878b-0c021779fac2-kube-api-access-qmhvn\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:30.273770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.273737 4312 generic.go:356] "Generic (PLEG): container finished" podID="602fd8f0-fea3-4c23-878b-0c021779fac2" containerID="75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598" exitCode=0 Nov 02 12:34:30.274191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.273789 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" Nov 02 12:34:30.274191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.273814 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" event={"ID":"602fd8f0-fea3-4c23-878b-0c021779fac2","Type":"ContainerDied","Data":"75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598"} Nov 02 12:34:30.274191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.273864 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr" event={"ID":"602fd8f0-fea3-4c23-878b-0c021779fac2","Type":"ContainerDied","Data":"6d50ffab1744d319a66e916ce80b27ac793d8c9c47e9541761f68b5258f3dfa2"} Nov 02 12:34:30.274191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.273886 4312 scope.go:122] "RemoveContainer" containerID="75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598" Nov 02 12:34:30.283271 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.283252 4312 scope.go:122] "RemoveContainer" containerID="75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598" Nov 02 12:34:30.283625 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:30.283605 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598\": container with ID starting with 75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598 not found: ID does not exist" containerID="75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598" Nov 02 12:34:30.283673 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.283633 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598"} err="failed to get container status \"75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598\": rpc error: code = NotFound desc = could not find container \"75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598\": container with ID starting with 75c2731c7ec83070de2dfcd4d582bdd01da26845ac32427162a198d1b7522598 not found: ID does not exist" Nov 02 12:34:30.438836 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.438810 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr"] Nov 02 12:34:30.488251 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:30.488227 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2e101a-575766cc75-zfhjr"] Nov 02 12:34:31.077388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:31.077354 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="602fd8f0-fea3-4c23-878b-0c021779fac2" path="/var/lib/kubelet/pods/602fd8f0-fea3-4c23-878b-0c021779fac2/volumes" Nov 02 12:34:31.159943 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:31.159910 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:31.401001 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:31.400919 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:31.690993 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:31.690964 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:32.138360 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:32.138285 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:33.260126 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.260094 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg"] Nov 02 12:34:33.262780 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.262749 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" podUID="10a368c1-41d7-41bf-a4b7-a36e38e0958d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 12:34:33.310205 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.310180 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w"] Nov 02 12:34:33.310590 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.310575 4312 cpu_manager.go:414] "containerMap: removing container" podUID="b54c847f-637c-4b25-b875-b2087f210293" containerName="payload-processing" Nov 02 12:34:33.310643 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.310591 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54c847f-637c-4b25-b875-b2087f210293" containerName="payload-processing" Nov 02 12:34:33.310643 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.310617 4312 cpu_manager.go:414] "containerMap: removing container" podUID="602fd8f0-fea3-4c23-878b-0c021779fac2" containerName="maas-api" Nov 02 12:34:33.310643 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.310622 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602fd8f0-fea3-4c23-878b-0c021779fac2" containerName="maas-api" Nov 02 12:34:33.310739 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.310679 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="602fd8f0-fea3-4c23-878b-0c021779fac2" containerName="maas-api" Nov 02 12:34:33.310739 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.310689 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54c847f-637c-4b25-b875-b2087f210293" containerName="payload-processing" Nov 02 12:34:33.315105 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.315088 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.374136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.374110 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w"] Nov 02 12:34:33.405280 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.405244 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxwhh\" (UniqueName: \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-kube-api-access-vxwhh\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.405420 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.405303 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.405420 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.405350 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.405578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.405489 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.506571 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.506542 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.506686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.506574 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.506686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.506609 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.506686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.506672 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxwhh\" (UniqueName: \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-kube-api-access-vxwhh\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.507299 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.507280 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.507341 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.507321 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.509077 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.509056 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.568052 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.567993 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxwhh\" (UniqueName: \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-kube-api-access-vxwhh\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.627938 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.627916 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:33.999707 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:33.999679 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w"] Nov 02 12:34:34.001677 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:34:34.001647 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4698fdca_2cfc_4210_aeab_e707895a54de.slice/crio-88b11cc0cbab01929b68760d22ee82eb4583be502e9930fe9b5b30d576244286 WatchSource:0}: Error finding container 88b11cc0cbab01929b68760d22ee82eb4583be502e9930fe9b5b30d576244286: Status 404 returned error can't find the container with id 88b11cc0cbab01929b68760d22ee82eb4583be502e9930fe9b5b30d576244286 Nov 02 12:34:34.302470 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:34.302376 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" event={"ID":"4698fdca-2cfc-4210-aeab-e707895a54de","Type":"ContainerStarted","Data":"173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877"} Nov 02 12:34:34.302888 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:34.302536 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:34.302888 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:34.302580 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" event={"ID":"4698fdca-2cfc-4210-aeab-e707895a54de","Type":"ContainerStarted","Data":"88b11cc0cbab01929b68760d22ee82eb4583be502e9930fe9b5b30d576244286"} Nov 02 12:34:34.418812 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:34.418755 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" podStartSLOduration=1.418737867 podStartE2EDuration="1.418737867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:34.417453043 +0000 UTC m=+2291.447689274" watchObservedRunningTime="2026-11-02 12:34:34.418737867 +0000 UTC m=+2291.448974117" Nov 02 12:34:34.755199 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:34.755166 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:35.087896 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:35.087775 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:36.403112 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:36.403036 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:36.799397 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:36.799360 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:37.047376 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:37.047345 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:37.717246 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:37.717217 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:38.260769 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.260731 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" podUID="10a368c1-41d7-41bf-a4b7-a36e38e0958d" containerName="payload-processing" containerID="cri-o://5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575" gracePeriod=25 Nov 02 12:34:38.325069 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.325045 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:38.505792 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.505764 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:38.552374 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.552323 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-kube-api-access-k4rd4\" (UniqueName: \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-kube-api-access-k4rd4\") pod \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " Nov 02 12:34:38.552374 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.552367 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-plugins-config-volume\") pod \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " Nov 02 12:34:38.552538 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.552457 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-provider-credentials\") pod \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " Nov 02 12:34:38.552538 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.552481 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-routing-overlay-volume\") pod \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\" (UID: \"10a368c1-41d7-41bf-a4b7-a36e38e0958d\") " Nov 02 12:34:38.552855 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.552828 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-plugins-config-volume" pod "10a368c1-41d7-41bf-a4b7-a36e38e0958d" (UID: "10a368c1-41d7-41bf-a4b7-a36e38e0958d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:34:38.552946 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.552878 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-routing-overlay-volume" pod "10a368c1-41d7-41bf-a4b7-a36e38e0958d" (UID: "10a368c1-41d7-41bf-a4b7-a36e38e0958d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:34:38.554568 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.554539 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-provider-credentials" pod "10a368c1-41d7-41bf-a4b7-a36e38e0958d" (UID: "10a368c1-41d7-41bf-a4b7-a36e38e0958d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:34:38.554568 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.554555 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-kube-api-access-k4rd4" pod "10a368c1-41d7-41bf-a4b7-a36e38e0958d" (UID: "10a368c1-41d7-41bf-a4b7-a36e38e0958d"). InnerVolumeSpecName "kube-api-access-k4rd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:34:38.584730 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.584681 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:38.655110 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.653983 4312 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-provider-credentials\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:38.655110 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.654011 4312 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-routing-overlay-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:38.655110 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.654030 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4rd4\" (UniqueName: \"kubernetes.io/projected/10a368c1-41d7-41bf-a4b7-a36e38e0958d-kube-api-access-k4rd4\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:38.655110 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.654046 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10a368c1-41d7-41bf-a4b7-a36e38e0958d-plugins-config-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:38.913167 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:38.913136 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:39.328102 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:39.328017 4312 generic.go:356] "Generic (PLEG): container finished" podID="10a368c1-41d7-41bf-a4b7-a36e38e0958d" containerID="5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575" exitCode=0 Nov 02 12:34:39.328102 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:39.328066 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" Nov 02 12:34:39.328339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:39.328103 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" event={"ID":"10a368c1-41d7-41bf-a4b7-a36e38e0958d","Type":"ContainerDied","Data":"5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575"} Nov 02 12:34:39.328339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:39.328171 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg" event={"ID":"10a368c1-41d7-41bf-a4b7-a36e38e0958d","Type":"ContainerDied","Data":"a6d564e8fd4bdf4167fd8634ffbcb4d89a2e97f8e35715528cf97b7c19692f7b"} Nov 02 12:34:39.328339 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:39.328202 4312 scope.go:122] "RemoveContainer" containerID="5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575" Nov 02 12:34:39.336459 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:39.336434 4312 scope.go:122] "RemoveContainer" containerID="5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575" Nov 02 12:34:39.336727 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:39.336708 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575\": container with ID starting with 5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575 not found: ID does not exist" containerID="5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575" Nov 02 12:34:39.336799 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:39.336738 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575"} err="failed to get container status \"5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575\": rpc error: code = NotFound desc = could not find container \"5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575\": container with ID starting with 5186615533ca19b887d790e4faf013511184057df97ee76b2394fe1107375575 not found: ID does not exist" Nov 02 12:34:39.491112 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:39.491086 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg"] Nov 02 12:34:39.540900 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:39.540876 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-66b5cjkrg"] Nov 02 12:34:39.896918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:39.896890 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:40.143599 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:40.143567 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:41.074194 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:41.074166 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a368c1-41d7-41bf-a4b7-a36e38e0958d" path="/var/lib/kubelet/pods/10a368c1-41d7-41bf-a4b7-a36e38e0958d/volumes" Nov 02 12:34:41.996556 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:41.996524 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:43.893182 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:43.893151 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:44.208137 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:44.208101 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-134e84d82424: secret "agc-cred-134e84d82424" not found Nov 02 12:34:44.208313 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:44.208150 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-dcd507907f10: secret "agc-cred-dcd507907f10" not found Nov 02 12:34:44.208313 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:44.208164 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr: [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:34:44.208313 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:44.208260 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials podName:91b75ed7-022a-4d07-9d7b-a699d52d087a nodeName:}" failed. No retries permitted until 2026-11-02 12:34:44.708239604 +0000 UTC m=+2301.738475853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" (UID: "91b75ed7-022a-4d07-9d7b-a699d52d087a") : [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:34:44.712453 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:44.712414 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-134e84d82424: secret "agc-cred-134e84d82424" not found Nov 02 12:34:44.712453 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:44.712456 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-dcd507907f10: secret "agc-cred-dcd507907f10" not found Nov 02 12:34:44.712730 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:44.712468 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr: [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:34:44.712730 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:44.712569 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials podName:91b75ed7-022a-4d07-9d7b-a699d52d087a nodeName:}" failed. No retries permitted until 2026-11-02 12:34:45.712547075 +0000 UTC m=+2302.742783323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" (UID: "91b75ed7-022a-4d07-9d7b-a699d52d087a") : [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:34:45.310428 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.310395 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" podUID="4698fdca-2cfc-4210-aeab-e707895a54de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 12:34:45.350863 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.350826 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w"] Nov 02 12:34:45.352097 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.352071 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" podUID="4698fdca-2cfc-4210-aeab-e707895a54de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 12:34:45.400285 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.400260 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc"] Nov 02 12:34:45.400681 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.400667 4312 cpu_manager.go:414] "containerMap: removing container" podUID="10a368c1-41d7-41bf-a4b7-a36e38e0958d" containerName="payload-processing" Nov 02 12:34:45.400742 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.400682 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a368c1-41d7-41bf-a4b7-a36e38e0958d" containerName="payload-processing" Nov 02 12:34:45.400787 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.400756 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a368c1-41d7-41bf-a4b7-a36e38e0958d" containerName="payload-processing" Nov 02 12:34:45.405095 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.405078 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.456816 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.456797 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-3a78b4\"/\"agc-cred-6433487f6bd7\"" Nov 02 12:34:45.467766 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.467748 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc"] Nov 02 12:34:45.520361 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.520335 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlfpn\" (UniqueName: \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-kube-api-access-xlfpn\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.520485 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.520379 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.520610 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.520589 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.520647 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.520621 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.621948 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.621869 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.621948 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.621911 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.622143 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.621966 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlfpn\" (UniqueName: \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-kube-api-access-xlfpn\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.622143 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.622012 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.622729 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.622704 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.622812 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.622773 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.624554 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.624532 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.682810 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.682784 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlfpn\" (UniqueName: \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-kube-api-access-xlfpn\") pod \"payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.718475 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.718450 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:45.722986 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:45.722967 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-134e84d82424: secret "agc-cred-134e84d82424" not found Nov 02 12:34:45.723068 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:45.723016 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-dcd507907f10: secret "agc-cred-dcd507907f10" not found Nov 02 12:34:45.723068 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:45.723029 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr: [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:34:45.723176 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:45.723090 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials podName:91b75ed7-022a-4d07-9d7b-a699d52d087a nodeName:}" failed. No retries permitted until 2026-11-02 12:34:47.723071664 +0000 UTC m=+2304.753307906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" (UID: "91b75ed7-022a-4d07-9d7b-a699d52d087a") : [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:34:45.823993 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:45.823945 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:46.092291 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:46.092271 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc"] Nov 02 12:34:46.094072 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:34:46.094048 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56122628_46a5_4439_bc4a_dbea61accc9c.slice/crio-22800108a05245d6ff38253c4eeff3aec26f6f40f19b4b71bf244c5009a9d511 WatchSource:0}: Error finding container 22800108a05245d6ff38253c4eeff3aec26f6f40f19b4b71bf244c5009a9d511: Status 404 returned error can't find the container with id 22800108a05245d6ff38253c4eeff3aec26f6f40f19b4b71bf244c5009a9d511 Nov 02 12:34:46.368597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:46.368507 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" event={"ID":"56122628-46a5-4439-bc4a-dbea61accc9c","Type":"ContainerStarted","Data":"968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6"} Nov 02 12:34:46.368597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:46.368564 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" event={"ID":"56122628-46a5-4439-bc4a-dbea61accc9c","Type":"ContainerStarted","Data":"22800108a05245d6ff38253c4eeff3aec26f6f40f19b4b71bf244c5009a9d511"} Nov 02 12:34:46.369027 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:46.368602 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:46.484617 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:46.484571 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" podStartSLOduration=1.484551618 podStartE2EDuration="1.484551618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:46.483693395 +0000 UTC m=+2303.513929670" watchObservedRunningTime="2026-11-02 12:34:46.484551618 +0000 UTC m=+2303.514787870" Nov 02 12:34:46.962907 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:46.962874 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:47.233238 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:47.233139 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:47.742994 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:47.742963 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-134e84d82424: secret "agc-cred-134e84d82424" not found Nov 02 12:34:47.742994 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:47.742996 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-dcd507907f10: secret "agc-cred-dcd507907f10" not found Nov 02 12:34:47.743547 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:47.743008 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr: [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:34:47.743547 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:47.743088 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials podName:91b75ed7-022a-4d07-9d7b-a699d52d087a nodeName:}" failed. No retries permitted until 2026-11-02 12:34:51.743067469 +0000 UTC m=+2308.773303714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" (UID: "91b75ed7-022a-4d07-9d7b-a699d52d087a") : [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:34:49.648801 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:49.648769 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:50.237743 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.237707 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:50.351489 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.351446 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" podUID="4698fdca-2cfc-4210-aeab-e707895a54de" containerName="payload-processing" containerID="cri-o://173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877" gracePeriod=25 Nov 02 12:34:50.487400 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.487365 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:50.606280 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.606253 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:50.774272 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.774195 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-provider-credentials\") pod \"4698fdca-2cfc-4210-aeab-e707895a54de\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " Nov 02 12:34:50.774272 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.774243 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-plugins-config-volume\") pod \"4698fdca-2cfc-4210-aeab-e707895a54de\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " Nov 02 12:34:50.774788 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.774292 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-routing-overlay-volume\") pod \"4698fdca-2cfc-4210-aeab-e707895a54de\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " Nov 02 12:34:50.774788 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.774384 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-kube-api-access-vxwhh\" (UniqueName: \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-kube-api-access-vxwhh\") pod \"4698fdca-2cfc-4210-aeab-e707895a54de\" (UID: \"4698fdca-2cfc-4210-aeab-e707895a54de\") " Nov 02 12:34:50.774788 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.774706 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-routing-overlay-volume" pod "4698fdca-2cfc-4210-aeab-e707895a54de" (UID: "4698fdca-2cfc-4210-aeab-e707895a54de"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:34:50.774788 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.774753 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-plugins-config-volume" pod "4698fdca-2cfc-4210-aeab-e707895a54de" (UID: "4698fdca-2cfc-4210-aeab-e707895a54de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:34:50.774936 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.774815 4312 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-routing-overlay-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:50.776294 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.776275 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-provider-credentials" pod "4698fdca-2cfc-4210-aeab-e707895a54de" (UID: "4698fdca-2cfc-4210-aeab-e707895a54de"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:34:50.776444 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.776426 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-kube-api-access-vxwhh" pod "4698fdca-2cfc-4210-aeab-e707895a54de" (UID: "4698fdca-2cfc-4210-aeab-e707895a54de"). InnerVolumeSpecName "kube-api-access-vxwhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:34:50.876153 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.876130 4312 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-provider-credentials\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:50.876153 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.876152 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4698fdca-2cfc-4210-aeab-e707895a54de-plugins-config-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:50.876290 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:50.876161 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxwhh\" (UniqueName: \"kubernetes.io/projected/4698fdca-2cfc-4210-aeab-e707895a54de-kube-api-access-vxwhh\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:34:51.393775 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:51.393696 4312 generic.go:356] "Generic (PLEG): container finished" podID="4698fdca-2cfc-4210-aeab-e707895a54de" containerID="173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877" exitCode=0 Nov 02 12:34:51.393775 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:51.393745 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" Nov 02 12:34:51.393963 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:51.393772 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" event={"ID":"4698fdca-2cfc-4210-aeab-e707895a54de","Type":"ContainerDied","Data":"173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877"} Nov 02 12:34:51.393963 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:51.393829 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w" event={"ID":"4698fdca-2cfc-4210-aeab-e707895a54de","Type":"ContainerDied","Data":"88b11cc0cbab01929b68760d22ee82eb4583be502e9930fe9b5b30d576244286"} Nov 02 12:34:51.393963 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:51.393856 4312 scope.go:122] "RemoveContainer" containerID="173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877" Nov 02 12:34:51.402117 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:51.402101 4312 scope.go:122] "RemoveContainer" containerID="173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877" Nov 02 12:34:51.402377 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:51.402359 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877\": container with ID starting with 173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877 not found: ID does not exist" containerID="173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877" Nov 02 12:34:51.402438 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:51.402382 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877"} err="failed to get container status \"173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877\": rpc error: code = NotFound desc = could not find container \"173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877\": container with ID starting with 173c165b0f21725445b639a20a33a7d03791e5162adcf6300e7bc1df6dfd5877 not found: ID does not exist" Nov 02 12:34:51.557173 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:51.557145 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w"] Nov 02 12:34:51.606553 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:51.606523 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-8696jbq7w"] Nov 02 12:34:51.782819 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:51.782795 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-134e84d82424: secret "agc-cred-134e84d82424" not found Nov 02 12:34:51.782819 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:51.782822 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-dcd507907f10: secret "agc-cred-dcd507907f10" not found Nov 02 12:34:51.783205 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:51.782831 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr: [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:34:51.783205 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:51.782878 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials podName:91b75ed7-022a-4d07-9d7b-a699d52d087a nodeName:}" failed. No retries permitted until 2026-11-02 12:34:59.782864235 +0000 UTC m=+2316.813100463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" (UID: "91b75ed7-022a-4d07-9d7b-a699d52d087a") : [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:34:52.092555 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:52.092467 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:53.074872 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:53.074841 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4698fdca-2cfc-4210-aeab-e707895a54de" path="/var/lib/kubelet/pods/4698fdca-2cfc-4210-aeab-e707895a54de/volumes" Nov 02 12:34:56.150691 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:56.150638 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:57.002124 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:57.002091 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:34:57.376319 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:57.376252 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:34:57.545055 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:34:57.545021 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr"] Nov 02 12:34:59.372686 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:59.372653 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:34:59.860621 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:59.860586 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-134e84d82424: secret "agc-cred-134e84d82424" not found Nov 02 12:34:59.860621 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:59.860624 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-dcd507907f10: secret "agc-cred-dcd507907f10" not found Nov 02 12:34:59.860892 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:59.860639 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr: [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:34:59.860892 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:34:59.860762 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials podName:91b75ed7-022a-4d07-9d7b-a699d52d087a nodeName:}" failed. No retries permitted until 2026-11-02 12:35:15.860741326 +0000 UTC m=+2332.890977568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" (UID: "91b75ed7-022a-4d07-9d7b-a699d52d087a") : [secret "agc-cred-134e84d82424" not found, secret "agc-cred-dcd507907f10" not found] Nov 02 12:35:01.797476 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:01.797434 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:02.545817 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:02.545761 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" podUID="91b75ed7-022a-4d07-9d7b-a699d52d087a" containerName="payload-processing" containerID="cri-o://dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931" gracePeriod=25 Nov 02 12:35:02.805071 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:02.805018 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:35:02.988891 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:02.988861 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-plugins-config-volume\") pod \"91b75ed7-022a-4d07-9d7b-a699d52d087a\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " Nov 02 12:35:02.989037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:02.988898 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-routing-overlay-volume\") pod \"91b75ed7-022a-4d07-9d7b-a699d52d087a\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " Nov 02 12:35:02.989037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:02.988938 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials\") pod \"91b75ed7-022a-4d07-9d7b-a699d52d087a\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " Nov 02 12:35:02.989037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:02.988980 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-kube-api-access-6rbg6\" (UniqueName: \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-kube-api-access-6rbg6\") pod \"91b75ed7-022a-4d07-9d7b-a699d52d087a\" (UID: \"91b75ed7-022a-4d07-9d7b-a699d52d087a\") " Nov 02 12:35:02.989320 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:02.989294 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-plugins-config-volume" pod "91b75ed7-022a-4d07-9d7b-a699d52d087a" (UID: "91b75ed7-022a-4d07-9d7b-a699d52d087a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:35:02.989320 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:02.989307 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-routing-overlay-volume" pod "91b75ed7-022a-4d07-9d7b-a699d52d087a" (UID: "91b75ed7-022a-4d07-9d7b-a699d52d087a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:35:02.991123 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:02.991095 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-kube-api-access-6rbg6" pod "91b75ed7-022a-4d07-9d7b-a699d52d087a" (UID: "91b75ed7-022a-4d07-9d7b-a699d52d087a"). InnerVolumeSpecName "kube-api-access-6rbg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:35:02.991219 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:02.991129 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials" pod "91b75ed7-022a-4d07-9d7b-a699d52d087a" (UID: "91b75ed7-022a-4d07-9d7b-a699d52d087a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:35:03.090172 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.090151 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-plugins-config-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:03.090172 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.090173 4312 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91b75ed7-022a-4d07-9d7b-a699d52d087a-routing-overlay-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:03.090305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.090183 4312 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-provider-credentials\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:03.090305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.090195 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rbg6\" (UniqueName: \"kubernetes.io/projected/91b75ed7-022a-4d07-9d7b-a699d52d087a-kube-api-access-6rbg6\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:03.459465 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.459435 4312 generic.go:356] "Generic (PLEG): container finished" podID="91b75ed7-022a-4d07-9d7b-a699d52d087a" containerID="dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931" exitCode=0 Nov 02 12:35:03.459625 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.459476 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" Nov 02 12:35:03.459625 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.459479 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" event={"ID":"91b75ed7-022a-4d07-9d7b-a699d52d087a","Type":"ContainerDied","Data":"dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931"} Nov 02 12:35:03.459625 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.459537 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr" event={"ID":"91b75ed7-022a-4d07-9d7b-a699d52d087a","Type":"ContainerDied","Data":"2d8192ade6fefb89a4337bdd31b558df8ba4d0e0cdb19b18f26203a6e8096f17"} Nov 02 12:35:03.459625 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.459558 4312 scope.go:122] "RemoveContainer" containerID="dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931" Nov 02 12:35:03.467735 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.467716 4312 scope.go:122] "RemoveContainer" containerID="dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931" Nov 02 12:35:03.468002 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:35:03.467985 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931\": container with ID starting with dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931 not found: ID does not exist" containerID="dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931" Nov 02 12:35:03.468072 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.468011 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931"} err="failed to get container status \"dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931\": rpc error: code = NotFound desc = could not find container \"dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931\": container with ID starting with dd6b1b0258a15ec9888d7943b30049fc5a7b8ac76f2dbcfdf81438902f652931 not found: ID does not exist" Nov 02 12:35:03.632988 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.632962 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr"] Nov 02 12:35:03.682565 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:03.682311 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-59bb4qhjr"] Nov 02 12:35:05.046305 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:05.046265 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94"] Nov 02 12:35:05.075002 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:05.074972 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b75ed7-022a-4d07-9d7b-a699d52d087a" path="/var/lib/kubelet/pods/91b75ed7-022a-4d07-9d7b-a699d52d087a/volumes" Nov 02 12:35:05.095668 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:05.095515 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba84ecdc-r2c94"] Nov 02 12:35:06.681599 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:06.681563 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:07.074385 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:07.074314 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de26b59a-f1d3-43a7-b504-7ccd797e80a3" path="/var/lib/kubelet/pods/de26b59a-f1d3-43a7-b504-7ccd797e80a3/volumes" Nov 02 12:35:09.175905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:09.175875 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:10.928959 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:10.928926 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:12.702873 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:12.702841 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:14.047192 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:14.047150 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg"] Nov 02 12:35:14.096331 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:14.096299 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ea176dd-725kg"] Nov 02 12:35:15.097166 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:15.097132 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3d444a-4cc6-41ea-945f-8e066d221bce" path="/var/lib/kubelet/pods/da3d444a-4cc6-41ea-945f-8e066d221bce/volumes" Nov 02 12:35:16.769178 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:16.769144 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:17.589163 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:17.589125 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:19.705167 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:19.705136 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:19.953638 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:19.953604 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:21.281272 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:21.281209 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:21.797298 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:21.797240 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:22.683445 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:22.683409 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:22.932575 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:22.932541 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:23.224768 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:23.224743 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:24.999391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:24.999338 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:27.919530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:27.919482 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq"] Nov 02 12:35:27.920084 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:27.920063 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4698fdca-2cfc-4210-aeab-e707895a54de" containerName="payload-processing" Nov 02 12:35:27.920169 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:27.920086 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4698fdca-2cfc-4210-aeab-e707895a54de" containerName="payload-processing" Nov 02 12:35:27.920169 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:27.920127 4312 cpu_manager.go:414] "containerMap: removing container" podUID="91b75ed7-022a-4d07-9d7b-a699d52d087a" containerName="payload-processing" Nov 02 12:35:27.920169 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:27.920135 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b75ed7-022a-4d07-9d7b-a699d52d087a" containerName="payload-processing" Nov 02 12:35:27.920324 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:27.920250 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4698fdca-2cfc-4210-aeab-e707895a54de" containerName="payload-processing" Nov 02 12:35:27.920324 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:27.920268 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b75ed7-022a-4d07-9d7b-a699d52d087a" containerName="payload-processing" Nov 02 12:35:27.924907 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:27.924887 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:27.989690 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:27.989657 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:27.990484 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:27.990459 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-6373b9\"/\"openshift-service-ca.crt\"" Nov 02 12:35:27.990615 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:27.990543 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-6373b9\"/\"kube-root-ca.crt\"" Nov 02 12:35:28.004172 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.004144 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-home\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.004276 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.004197 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jmlx\" (UniqueName: \"kubernetes.io/projected/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-kube-api-access-4jmlx\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.004320 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.004270 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-model-cache\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.004320 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.004313 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-dshm\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.004401 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.004346 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tls-certs\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.004537 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.004518 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tmp-dir\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.029508 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.029480 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-6373b9\"/\"default-dockercfg-hxkbs\"" Nov 02 12:35:28.029604 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.029539 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-6373b9\"/\"explicit-model-6373b9-kserve-self-signed-certs\"" Nov 02 12:35:28.031579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.031560 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq"] Nov 02 12:35:28.105883 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.105856 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jmlx\" (UniqueName: \"kubernetes.io/projected/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-kube-api-access-4jmlx\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.105994 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.105897 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-model-cache\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.105994 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.105936 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-dshm\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.105994 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.105978 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tls-certs\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.106163 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.106040 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tmp-dir\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.106163 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.106105 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-home\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.106438 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.106414 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-model-cache\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.106534 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.106441 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-home\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.106534 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.106465 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tmp-dir\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.108171 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.108151 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-dshm\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.108437 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.108422 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tls-certs\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.166371 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.166348 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jmlx\" (UniqueName: \"kubernetes.io/projected/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-kube-api-access-4jmlx\") pod \"explicit-model-6373b9-kserve-77c664b4f5-s49bq\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.233385 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.233314 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:28.238234 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.238201 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:28.615365 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:28.615321 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq"] Nov 02 12:35:28.617629 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:35:28.617604 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc670745e_5302_4d5d_b3a6_2d5d2b5b00f7.slice/crio-b50eadeab2d5b706edb48b91244d74b2841fc9cf02c57221164585c037af68e6 WatchSource:0}: Error finding container b50eadeab2d5b706edb48b91244d74b2841fc9cf02c57221164585c037af68e6: Status 404 returned error can't find the container with id b50eadeab2d5b706edb48b91244d74b2841fc9cf02c57221164585c037af68e6 Nov 02 12:35:29.307474 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:29.307437 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:29.372158 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:35:29.372122 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:35:29.556646 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:29.556620 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:29.597755 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:29.597691 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" event={"ID":"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7","Type":"ContainerStarted","Data":"1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1"} Nov 02 12:35:29.597755 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:29.597729 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" event={"ID":"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7","Type":"ContainerStarted","Data":"b50eadeab2d5b706edb48b91244d74b2841fc9cf02c57221164585c037af68e6"} Nov 02 12:35:29.715113 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:29.715071 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" podStartSLOduration=2.71505915 podStartE2EDuration="2.71505915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:29.713586727 +0000 UTC m=+2346.743822978" watchObservedRunningTime="2026-11-02 12:35:29.71505915 +0000 UTC m=+2346.745295399" Nov 02 12:35:29.960801 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:29.960744 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:30.238604 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:30.238540 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:32.983100 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:32.983068 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:36.298940 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:36.298900 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:36.558310 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:36.558237 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:37.890136 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:37.890102 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:38.238603 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:38.238572 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:38.251206 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:38.251180 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:38.653155 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:38.653071 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:35:41.126050 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:41.126010 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj"] Nov 02 12:35:41.126535 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:41.126294 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" podUID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" containerName="istio-proxy" containerID="cri-o://fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c" gracePeriod=30 Nov 02 12:35:42.790799 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:42.790753 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:45.412461 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:45.412430 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 12:35:45.412872 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:45.412479 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" podUID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 12:35:46.368231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.368207 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:35:46.469432 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469409 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-certs\") pod \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " Nov 02 12:35:46.469886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469486 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-kube-api-access-8pkzc\" (UniqueName: \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-kube-api-access-8pkzc\") pod \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " Nov 02 12:35:46.469886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469536 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-data\") pod \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " Nov 02 12:35:46.469886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469561 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-socket\") pod \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " Nov 02 12:35:46.469886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469614 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-envoy\") pod \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " Nov 02 12:35:46.469886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469643 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-podinfo\") pod \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " Nov 02 12:35:46.469886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469692 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-credential-socket\") pod \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " Nov 02 12:35:46.469886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469710 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-certs" pod "72aa094f-f655-4b75-8a90-f1a2536ce9d6" (UID: "72aa094f-f655-4b75-8a90-f1a2536ce9d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:35:46.469886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469747 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istiod-ca-cert\") pod \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " Nov 02 12:35:46.469886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469782 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-token\" (UniqueName: \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-token\") pod \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\" (UID: \"72aa094f-f655-4b75-8a90-f1a2536ce9d6\") " Nov 02 12:35:46.470372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.470139 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:46.470372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469780 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-data" pod "72aa094f-f655-4b75-8a90-f1a2536ce9d6" (UID: "72aa094f-f655-4b75-8a90-f1a2536ce9d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:35:46.470372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.469991 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-socket" pod "72aa094f-f655-4b75-8a90-f1a2536ce9d6" (UID: "72aa094f-f655-4b75-8a90-f1a2536ce9d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:35:46.470372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.470104 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istiod-ca-cert" pod "72aa094f-f655-4b75-8a90-f1a2536ce9d6" (UID: "72aa094f-f655-4b75-8a90-f1a2536ce9d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:35:46.470372 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.470187 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-credential-socket" pod "72aa094f-f655-4b75-8a90-f1a2536ce9d6" (UID: "72aa094f-f655-4b75-8a90-f1a2536ce9d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:35:46.471918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.471893 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-kube-api-access-8pkzc" pod "72aa094f-f655-4b75-8a90-f1a2536ce9d6" (UID: "72aa094f-f655-4b75-8a90-f1a2536ce9d6"). InnerVolumeSpecName "kube-api-access-8pkzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:35:46.472024 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.471912 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-podinfo" pod "72aa094f-f655-4b75-8a90-f1a2536ce9d6" (UID: "72aa094f-f655-4b75-8a90-f1a2536ce9d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:35:46.472024 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.471926 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-token" pod "72aa094f-f655-4b75-8a90-f1a2536ce9d6" (UID: "72aa094f-f655-4b75-8a90-f1a2536ce9d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:35:46.472024 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.471938 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-envoy" pod "72aa094f-f655-4b75-8a90-f1a2536ce9d6" (UID: "72aa094f-f655-4b75-8a90-f1a2536ce9d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:35:46.570929 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.570899 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pkzc\" (UniqueName: \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-kube-api-access-8pkzc\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:46.570929 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.570925 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:46.570929 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.570935 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:46.571125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.570943 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:46.571125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.570951 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:46.571125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.570960 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72aa094f-f655-4b75-8a90-f1a2536ce9d6-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:46.571125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.570969 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:46.571125 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.570978 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72aa094f-f655-4b75-8a90-f1a2536ce9d6-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:35:46.682555 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.682529 4312 generic.go:356] "Generic (PLEG): container finished" podID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" containerID="fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c" exitCode=0 Nov 02 12:35:46.682686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.682582 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" Nov 02 12:35:46.682686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.682610 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" event={"ID":"72aa094f-f655-4b75-8a90-f1a2536ce9d6","Type":"ContainerDied","Data":"fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c"} Nov 02 12:35:46.682686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.682659 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj" event={"ID":"72aa094f-f655-4b75-8a90-f1a2536ce9d6","Type":"ContainerDied","Data":"bac0e7007070503dccca5d31ceb0af1e5a9b36e0e1265be417b14ed8cc3c1105"} Nov 02 12:35:46.682686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.682680 4312 scope.go:122] "RemoveContainer" containerID="fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c" Nov 02 12:35:46.693852 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.693836 4312 scope.go:122] "RemoveContainer" containerID="fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c" Nov 02 12:35:46.694158 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:35:46.694141 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c\": container with ID starting with fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c not found: ID does not exist" containerID="fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c" Nov 02 12:35:46.694215 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.694167 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c"} err="failed to get container status \"fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c\": rpc error: code = NotFound desc = could not find container \"fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c\": container with ID starting with fb18ebad60aa6e56eb4e1d6781cbc908a9b32337ba4bab8b5180b27ba60bfe5c not found: ID does not exist" Nov 02 12:35:46.848354 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.848282 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj"] Nov 02 12:35:46.852586 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:46.852563 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-cp4gj"] Nov 02 12:35:47.075664 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:47.075637 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" path="/var/lib/kubelet/pods/72aa094f-f655-4b75-8a90-f1a2536ce9d6/volumes" Nov 02 12:35:47.697592 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:47.697554 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:48.901566 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:48.901534 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq"] Nov 02 12:35:49.035156 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:49.035126 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:49.475266 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:49.475232 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:52.581063 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:52.581034 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:54.497175 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:54.497140 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:57.490883 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:35:57.490851 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:35:59.374013 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:35:59.373965 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:36:02.382868 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:02.382836 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:03.920893 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:03.920858 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" podUID="c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" containerName="main" containerID="cri-o://1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1" gracePeriod=45 Nov 02 12:36:04.169972 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.169949 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:36:04.329896 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.329839 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tls-certs\") pod \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " Nov 02 12:36:04.329896 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.329872 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-dshm\") pod \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " Nov 02 12:36:04.330066 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.329921 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-home\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-home\") pod \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " Nov 02 12:36:04.330066 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.329986 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tmp-dir\") pod \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " Nov 02 12:36:04.330066 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.330045 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-model-cache\") pod \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " Nov 02 12:36:04.330220 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.330076 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-kube-api-access-4jmlx\" (UniqueName: \"kubernetes.io/projected/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-kube-api-access-4jmlx\") pod \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\" (UID: \"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7\") " Nov 02 12:36:04.330285 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.330227 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-home" pod "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" (UID: "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:04.330285 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.330256 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tmp-dir" pod "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" (UID: "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:04.330410 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.330378 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-model-cache" pod "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" (UID: "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:04.330549 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.330472 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:04.330549 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.330491 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:04.332088 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.332071 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-dshm" pod "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" (UID: "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:04.332185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.332166 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tls-certs" pod "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" (UID: "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:36:04.332224 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.332201 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-kube-api-access-4jmlx" pod "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" (UID: "c670745e-5302-4d5d-b3a6-2d5d2b5b00f7"). InnerVolumeSpecName "kube-api-access-4jmlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:36:04.431041 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.431011 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:04.431041 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.431037 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:04.431194 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.431050 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:04.431194 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.431066 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jmlx\" (UniqueName: \"kubernetes.io/projected/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7-kube-api-access-4jmlx\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:04.773022 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.772982 4312 generic.go:356] "Generic (PLEG): container finished" podID="c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" containerID="1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1" exitCode=0 Nov 02 12:36:04.773190 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.773030 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" Nov 02 12:36:04.773190 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.773064 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" event={"ID":"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7","Type":"ContainerDied","Data":"1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1"} Nov 02 12:36:04.773190 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.773111 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq" event={"ID":"c670745e-5302-4d5d-b3a6-2d5d2b5b00f7","Type":"ContainerDied","Data":"b50eadeab2d5b706edb48b91244d74b2841fc9cf02c57221164585c037af68e6"} Nov 02 12:36:04.773190 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.773129 4312 scope.go:122] "RemoveContainer" containerID="1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1" Nov 02 12:36:04.782492 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.782475 4312 scope.go:122] "RemoveContainer" containerID="1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1" Nov 02 12:36:04.782870 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:36:04.782849 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1\": container with ID starting with 1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1 not found: ID does not exist" containerID="1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1" Nov 02 12:36:04.782921 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.782878 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1"} err="failed to get container status \"1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1\": rpc error: code = NotFound desc = could not find container \"1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1\": container with ID starting with 1c12e7d6cddb655354680e4f6a159d6acadce9f51788cee4822ab41933a81bf1 not found: ID does not exist" Nov 02 12:36:04.938959 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.938924 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq"] Nov 02 12:36:04.945088 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:04.945063 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-6373b9/explicit-model-6373b9-kserve-77c664b4f5-s49bq"] Nov 02 12:36:05.075074 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:05.075004 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" path="/var/lib/kubelet/pods/c670745e-5302-4d5d-b3a6-2d5d2b5b00f7/volumes" Nov 02 12:36:07.298785 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:07.298639 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:08.858241 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:08.858194 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:11.301777 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:11.301734 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45"] Nov 02 12:36:11.302274 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:11.301967 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" containerID="cri-o://d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6" gracePeriod=30 Nov 02 12:36:11.351598 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:11.351571 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z"] Nov 02 12:36:11.351746 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:11.351728 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" containerID="cri-o://4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6" gracePeriod=30 Nov 02 12:36:12.189005 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:12.188972 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:13.627771 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:13.627735 4312 patch_prober.go:29] interesting pod/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 12:36:13.628162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:13.627801 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 12:36:15.736775 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:15.736728 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mxd45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 12:36:15.737183 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:15.736789 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 12:36:16.596285 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.596262 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:36:16.624301 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.624280 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:36:16.626475 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626454 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-certs\") pod \"8a75c277-13b3-417e-a5b3-134cae493061\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " Nov 02 12:36:16.626599 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626486 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-socket\") pod \"eb392c56-19f5-4737-8846-7bf9b0911f1e\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " Nov 02 12:36:16.626599 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626525 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-credential-socket\") pod \"8a75c277-13b3-417e-a5b3-134cae493061\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " Nov 02 12:36:16.626599 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626547 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-istio-token\" (UniqueName: \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-istio-token\") pod \"8a75c277-13b3-417e-a5b3-134cae493061\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " Nov 02 12:36:16.626776 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626613 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-credential-socket\") pod \"eb392c56-19f5-4737-8846-7bf9b0911f1e\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " Nov 02 12:36:16.626776 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626642 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-kube-api-access-lhh6g\" (UniqueName: \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-kube-api-access-lhh6g\") pod \"eb392c56-19f5-4737-8846-7bf9b0911f1e\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " Nov 02 12:36:16.626776 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626677 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-data\") pod \"eb392c56-19f5-4737-8846-7bf9b0911f1e\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " Nov 02 12:36:16.626776 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626719 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a75c277-13b3-417e-a5b3-134cae493061-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a75c277-13b3-417e-a5b3-134cae493061-istio-podinfo\") pod \"8a75c277-13b3-417e-a5b3-134cae493061\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " Nov 02 12:36:16.626776 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626748 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-certs\") pod \"eb392c56-19f5-4737-8846-7bf9b0911f1e\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626770 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-socket" pod "eb392c56-19f5-4737-8846-7bf9b0911f1e" (UID: "eb392c56-19f5-4737-8846-7bf9b0911f1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626790 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-envoy\") pod \"eb392c56-19f5-4737-8846-7bf9b0911f1e\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626798 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-credential-socket" pod "8a75c277-13b3-417e-a5b3-134cae493061" (UID: "8a75c277-13b3-417e-a5b3-134cae493061"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626816 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-kube-api-access-jrh22\" (UniqueName: \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-kube-api-access-jrh22\") pod \"8a75c277-13b3-417e-a5b3-134cae493061\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626886 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb392c56-19f5-4737-8846-7bf9b0911f1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb392c56-19f5-4737-8846-7bf9b0911f1e-istiod-ca-cert\") pod \"eb392c56-19f5-4737-8846-7bf9b0911f1e\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626917 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-data\") pod \"8a75c277-13b3-417e-a5b3-134cae493061\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626947 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-envoy\") pod \"8a75c277-13b3-417e-a5b3-134cae493061\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626934 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-credential-socket" pod "eb392c56-19f5-4737-8846-7bf9b0911f1e" (UID: "eb392c56-19f5-4737-8846-7bf9b0911f1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626970 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-data" pod "eb392c56-19f5-4737-8846-7bf9b0911f1e" (UID: "eb392c56-19f5-4737-8846-7bf9b0911f1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626984 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-podinfo\") pod \"eb392c56-19f5-4737-8846-7bf9b0911f1e\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.626993 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-certs" pod "eb392c56-19f5-4737-8846-7bf9b0911f1e" (UID: "eb392c56-19f5-4737-8846-7bf9b0911f1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:16.627037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627023 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a75c277-13b3-417e-a5b3-134cae493061-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a75c277-13b3-417e-a5b3-134cae493061-istiod-ca-cert\") pod \"8a75c277-13b3-417e-a5b3-134cae493061\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " Nov 02 12:36:16.627902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627076 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-socket\") pod \"8a75c277-13b3-417e-a5b3-134cae493061\" (UID: \"8a75c277-13b3-417e-a5b3-134cae493061\") " Nov 02 12:36:16.627902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627122 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-token\" (UniqueName: \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-token\") pod \"eb392c56-19f5-4737-8846-7bf9b0911f1e\" (UID: \"eb392c56-19f5-4737-8846-7bf9b0911f1e\") " Nov 02 12:36:16.627902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627149 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-data" pod "8a75c277-13b3-417e-a5b3-134cae493061" (UID: "8a75c277-13b3-417e-a5b3-134cae493061"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:16.627902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627260 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb392c56-19f5-4737-8846-7bf9b0911f1e-istiod-ca-cert" pod "eb392c56-19f5-4737-8846-7bf9b0911f1e" (UID: "eb392c56-19f5-4737-8846-7bf9b0911f1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:36:16.627902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627384 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-socket" pod "8a75c277-13b3-417e-a5b3-134cae493061" (UID: "8a75c277-13b3-417e-a5b3-134cae493061"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:16.627902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627453 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a75c277-13b3-417e-a5b3-134cae493061-istiod-ca-cert" pod "8a75c277-13b3-417e-a5b3-134cae493061" (UID: "8a75c277-13b3-417e-a5b3-134cae493061"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:36:16.627902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627553 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-certs" pod "8a75c277-13b3-417e-a5b3-134cae493061" (UID: "8a75c277-13b3-417e-a5b3-134cae493061"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:16.627902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627852 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.627902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627876 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.627902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627890 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.627902 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627904 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.628509 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627917 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.628509 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627930 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.628509 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627944 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb392c56-19f5-4737-8846-7bf9b0911f1e-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.628509 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.627958 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.628509 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.628002 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a75c277-13b3-417e-a5b3-134cae493061-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.628509 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.628016 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.629376 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.629335 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a75c277-13b3-417e-a5b3-134cae493061-istio-podinfo" pod "8a75c277-13b3-417e-a5b3-134cae493061" (UID: "8a75c277-13b3-417e-a5b3-134cae493061"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:36:16.629516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.629469 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-istio-token" pod "8a75c277-13b3-417e-a5b3-134cae493061" (UID: "8a75c277-13b3-417e-a5b3-134cae493061"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:36:16.629602 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.629538 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-token" pod "eb392c56-19f5-4737-8846-7bf9b0911f1e" (UID: "eb392c56-19f5-4737-8846-7bf9b0911f1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:36:16.629602 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.629546 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-envoy" pod "eb392c56-19f5-4737-8846-7bf9b0911f1e" (UID: "eb392c56-19f5-4737-8846-7bf9b0911f1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:16.629781 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.629765 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-kube-api-access-jrh22" pod "8a75c277-13b3-417e-a5b3-134cae493061" (UID: "8a75c277-13b3-417e-a5b3-134cae493061"). InnerVolumeSpecName "kube-api-access-jrh22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:36:16.630224 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.630208 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-kube-api-access-lhh6g" pod "eb392c56-19f5-4737-8846-7bf9b0911f1e" (UID: "eb392c56-19f5-4737-8846-7bf9b0911f1e"). InnerVolumeSpecName "kube-api-access-lhh6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:36:16.630422 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.630409 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-envoy" pod "8a75c277-13b3-417e-a5b3-134cae493061" (UID: "8a75c277-13b3-417e-a5b3-134cae493061"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:16.630741 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.630723 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-podinfo" pod "eb392c56-19f5-4737-8846-7bf9b0911f1e" (UID: "eb392c56-19f5-4737-8846-7bf9b0911f1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:36:16.728773 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.728748 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.728773 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.728768 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.728898 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.728778 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhh6g\" (UniqueName: \"kubernetes.io/projected/eb392c56-19f5-4737-8846-7bf9b0911f1e-kube-api-access-lhh6g\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.728898 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.728789 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a75c277-13b3-417e-a5b3-134cae493061-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.728898 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.728797 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.728898 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.728806 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrh22\" (UniqueName: \"kubernetes.io/projected/8a75c277-13b3-417e-a5b3-134cae493061-kube-api-access-jrh22\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.728898 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.728814 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a75c277-13b3-417e-a5b3-134cae493061-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.728898 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.728822 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb392c56-19f5-4737-8846-7bf9b0911f1e-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:16.836225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.836139 4312 generic.go:356] "Generic (PLEG): container finished" podID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerID="4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6" exitCode=0 Nov 02 12:36:16.836225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.836201 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" Nov 02 12:36:16.836805 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.836217 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" event={"ID":"eb392c56-19f5-4737-8846-7bf9b0911f1e","Type":"ContainerDied","Data":"4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6"} Nov 02 12:36:16.836805 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.836306 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z" event={"ID":"eb392c56-19f5-4737-8846-7bf9b0911f1e","Type":"ContainerDied","Data":"64ffa716d0a9680e0832cdddad6f336b4ace1d4ee2cf90ef763fb2715628dbde"} Nov 02 12:36:16.836805 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.836359 4312 scope.go:122] "RemoveContainer" containerID="4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6" Nov 02 12:36:16.837806 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.837785 4312 generic.go:356] "Generic (PLEG): container finished" podID="8a75c277-13b3-417e-a5b3-134cae493061" containerID="d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6" exitCode=0 Nov 02 12:36:16.837930 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.837816 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" event={"ID":"8a75c277-13b3-417e-a5b3-134cae493061","Type":"ContainerDied","Data":"d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6"} Nov 02 12:36:16.837930 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.837845 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" Nov 02 12:36:16.837930 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.837853 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45" event={"ID":"8a75c277-13b3-417e-a5b3-134cae493061","Type":"ContainerDied","Data":"26ed42da6229353de2681c9f1083fb4fb3a6de57e5d6e717f779b1cf6b3638a9"} Nov 02 12:36:16.847729 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.847706 4312 scope.go:122] "RemoveContainer" containerID="4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6" Nov 02 12:36:16.848009 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:36:16.847992 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6\": container with ID starting with 4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6 not found: ID does not exist" containerID="4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6" Nov 02 12:36:16.848065 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.848018 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6"} err="failed to get container status \"4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6\": rpc error: code = NotFound desc = could not find container \"4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6\": container with ID starting with 4f6401a4886c8fa2bec0c5feeff8cdfc94cb5b61068adbddd5ab0d76c648efb6 not found: ID does not exist" Nov 02 12:36:16.848065 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.848033 4312 scope.go:122] "RemoveContainer" containerID="d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6" Nov 02 12:36:16.855899 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.855882 4312 scope.go:122] "RemoveContainer" containerID="d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6" Nov 02 12:36:16.856154 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:36:16.856134 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6\": container with ID starting with d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6 not found: ID does not exist" containerID="d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6" Nov 02 12:36:16.856242 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.856159 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6"} err="failed to get container status \"d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6\": rpc error: code = NotFound desc = could not find container \"d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6\": container with ID starting with d2e953dbeb2220b6b5c11df68b54ce564b1f2681994ebc9c9131b97ca571a1f6 not found: ID does not exist" Nov 02 12:36:16.998822 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:16.998787 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45"] Nov 02 12:36:17.003647 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:17.003620 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mxd45"] Nov 02 12:36:17.075468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:17.075441 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a75c277-13b3-417e-a5b3-134cae493061" path="/var/lib/kubelet/pods/8a75c277-13b3-417e-a5b3-134cae493061/volumes" Nov 02 12:36:17.085944 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:17.085918 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:17.127377 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:17.127319 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z"] Nov 02 12:36:17.131578 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:17.131558 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-dd087a-openshift-default-5b6fb74d8c-dqd9z"] Nov 02 12:36:17.337996 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:17.337967 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:19.075035 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:19.074998 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" path="/var/lib/kubelet/pods/eb392c56-19f5-4737-8846-7bf9b0911f1e/volumes" Nov 02 12:36:21.974023 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:21.973990 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:23.574228 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:23.574192 4312 scope.go:122] "RemoveContainer" containerID="fa9fc0e7fd78b3ee5f61935289f228337055773eace90afa1538028d52071bdf" Nov 02 12:36:23.583409 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:23.583385 4312 scope.go:122] "RemoveContainer" containerID="0b1932931eff7ceeaca49b592ba330f66b558c420db734cbb3bcc2fb0c0f179b" Nov 02 12:36:26.821998 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:26.821966 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:29.372701 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:36:29.372669 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:36:31.468959 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:31.468923 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:31.716266 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:31.716217 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:32.597435 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:32.597405 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:32.845470 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:32.845432 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:36.872711 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:36.872679 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:37.409180 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:37.409144 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:38.123794 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:38.123763 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx"] Nov 02 12:36:38.124206 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:38.123959 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" podUID="6831a2f5-a9b7-4c08-a825-4c80554c601c" containerName="istio-proxy" containerID="cri-o://876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438" gracePeriod=30 Nov 02 12:36:41.774253 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:41.774225 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:42.411036 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:42.411006 4312 patch_prober.go:29] interesting pod/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 12:36:42.411197 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:42.411054 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" podUID="6831a2f5-a9b7-4c08-a825-4c80554c601c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 12:36:43.373209 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.373189 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:36:43.456665 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.456640 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-token\" (UniqueName: \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-token\") pod \"6831a2f5-a9b7-4c08-a825-4c80554c601c\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " Nov 02 12:36:43.456779 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.456679 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-socket\") pod \"6831a2f5-a9b7-4c08-a825-4c80554c601c\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " Nov 02 12:36:43.456779 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.456715 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-envoy\") pod \"6831a2f5-a9b7-4c08-a825-4c80554c601c\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " Nov 02 12:36:43.456779 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.456750 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-podinfo\") pod \"6831a2f5-a9b7-4c08-a825-4c80554c601c\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " Nov 02 12:36:43.456779 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.456775 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-credential-socket\") pod \"6831a2f5-a9b7-4c08-a825-4c80554c601c\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " Nov 02 12:36:43.456986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.456801 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-kube-api-access-7b7ld\" (UniqueName: \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-kube-api-access-7b7ld\") pod \"6831a2f5-a9b7-4c08-a825-4c80554c601c\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " Nov 02 12:36:43.456986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.456817 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-certs\") pod \"6831a2f5-a9b7-4c08-a825-4c80554c601c\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " Nov 02 12:36:43.456986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.456855 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6831a2f5-a9b7-4c08-a825-4c80554c601c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6831a2f5-a9b7-4c08-a825-4c80554c601c-istiod-ca-cert\") pod \"6831a2f5-a9b7-4c08-a825-4c80554c601c\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " Nov 02 12:36:43.456986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.456975 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-socket" pod "6831a2f5-a9b7-4c08-a825-4c80554c601c" (UID: "6831a2f5-a9b7-4c08-a825-4c80554c601c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:43.457188 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.457053 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-data\") pod \"6831a2f5-a9b7-4c08-a825-4c80554c601c\" (UID: \"6831a2f5-a9b7-4c08-a825-4c80554c601c\") " Nov 02 12:36:43.457188 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.457098 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-credential-socket" pod "6831a2f5-a9b7-4c08-a825-4c80554c601c" (UID: "6831a2f5-a9b7-4c08-a825-4c80554c601c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:43.457188 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.457175 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-certs" pod "6831a2f5-a9b7-4c08-a825-4c80554c601c" (UID: "6831a2f5-a9b7-4c08-a825-4c80554c601c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:43.457424 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.457373 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-data" pod "6831a2f5-a9b7-4c08-a825-4c80554c601c" (UID: "6831a2f5-a9b7-4c08-a825-4c80554c601c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:43.457533 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.457473 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:43.457533 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.457508 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:43.457533 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.457525 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:43.457695 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.457539 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:43.457695 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.457562 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6831a2f5-a9b7-4c08-a825-4c80554c601c-istiod-ca-cert" pod "6831a2f5-a9b7-4c08-a825-4c80554c601c" (UID: "6831a2f5-a9b7-4c08-a825-4c80554c601c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:36:43.458918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.458891 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-token" pod "6831a2f5-a9b7-4c08-a825-4c80554c601c" (UID: "6831a2f5-a9b7-4c08-a825-4c80554c601c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:36:43.459020 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.458928 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-kube-api-access-7b7ld" pod "6831a2f5-a9b7-4c08-a825-4c80554c601c" (UID: "6831a2f5-a9b7-4c08-a825-4c80554c601c"). InnerVolumeSpecName "kube-api-access-7b7ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:36:43.459020 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.458976 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-podinfo" pod "6831a2f5-a9b7-4c08-a825-4c80554c601c" (UID: "6831a2f5-a9b7-4c08-a825-4c80554c601c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:36:43.459020 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.458987 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-envoy" pod "6831a2f5-a9b7-4c08-a825-4c80554c601c" (UID: "6831a2f5-a9b7-4c08-a825-4c80554c601c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:36:43.558685 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.558664 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:43.558685 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.558685 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:43.558815 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.558695 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6831a2f5-a9b7-4c08-a825-4c80554c601c-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:43.558815 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.558705 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b7ld\" (UniqueName: \"kubernetes.io/projected/6831a2f5-a9b7-4c08-a825-4c80554c601c-kube-api-access-7b7ld\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:43.558815 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.558714 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6831a2f5-a9b7-4c08-a825-4c80554c601c-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:36:43.969457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.969417 4312 generic.go:356] "Generic (PLEG): container finished" podID="6831a2f5-a9b7-4c08-a825-4c80554c601c" containerID="876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438" exitCode=0 Nov 02 12:36:43.969649 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.969483 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" Nov 02 12:36:43.969649 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.969521 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" event={"ID":"6831a2f5-a9b7-4c08-a825-4c80554c601c","Type":"ContainerDied","Data":"876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438"} Nov 02 12:36:43.969649 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.969599 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx" event={"ID":"6831a2f5-a9b7-4c08-a825-4c80554c601c","Type":"ContainerDied","Data":"7de8fc932e61431c6d58799bfef78a80fe83d871436d4c7628eb2844724dbe27"} Nov 02 12:36:43.969649 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.969625 4312 scope.go:122] "RemoveContainer" containerID="876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438" Nov 02 12:36:43.978870 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.978848 4312 scope.go:122] "RemoveContainer" containerID="876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438" Nov 02 12:36:43.979151 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:36:43.979129 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438\": container with ID starting with 876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438 not found: ID does not exist" containerID="876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438" Nov 02 12:36:43.979208 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:43.979159 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438"} err="failed to get container status \"876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438\": rpc error: code = NotFound desc = could not find container \"876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438\": container with ID starting with 876aa23529d9a01919e163e7030ed05cafe337325926001723da9b9adab59438 not found: ID does not exist" Nov 02 12:36:44.133482 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:44.133458 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx"] Nov 02 12:36:44.182779 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:44.182758 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-03c4e8-openshift-default-6f858d7b75-bv5jx"] Nov 02 12:36:45.074076 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:45.074042 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6831a2f5-a9b7-4c08-a825-4c80554c601c" path="/var/lib/kubelet/pods/6831a2f5-a9b7-4c08-a825-4c80554c601c/volumes" Nov 02 12:36:46.021555 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:46.021522 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:47.784745 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:47.784709 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:50.616153 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:50.616109 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:52.231569 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:52.231533 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:52.484784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:52.484711 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:52.743277 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:52.743205 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:53.832490 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:53.832456 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:54.512659 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:54.512627 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:55.197802 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:55.197758 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:55.840948 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:55.840917 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:56.217283 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:56.217249 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:57.530325 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:57.530292 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:57.814154 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:57.814068 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:58.080449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:36:58.080373 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:36:59.372090 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:36:59.372057 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:37:00.706717 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.706686 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c"] Nov 02 12:37:00.707162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707046 4312 cpu_manager.go:414] "containerMap: removing container" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" Nov 02 12:37:00.707162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707056 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" Nov 02 12:37:00.707162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707075 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6831a2f5-a9b7-4c08-a825-4c80554c601c" containerName="istio-proxy" Nov 02 12:37:00.707162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707081 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6831a2f5-a9b7-4c08-a825-4c80554c601c" containerName="istio-proxy" Nov 02 12:37:00.707162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707092 4312 cpu_manager.go:414] "containerMap: removing container" podUID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" containerName="istio-proxy" Nov 02 12:37:00.707162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707097 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" containerName="istio-proxy" Nov 02 12:37:00.707162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707104 4312 cpu_manager.go:414] "containerMap: removing container" podUID="c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" containerName="main" Nov 02 12:37:00.707162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707108 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" containerName="main" Nov 02 12:37:00.707162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707115 4312 cpu_manager.go:414] "containerMap: removing container" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" Nov 02 12:37:00.707162 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707120 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" Nov 02 12:37:00.707620 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707196 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a75c277-13b3-417e-a5b3-134cae493061" containerName="istio-proxy" Nov 02 12:37:00.707620 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707205 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6831a2f5-a9b7-4c08-a825-4c80554c601c" containerName="istio-proxy" Nov 02 12:37:00.707620 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707211 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="72aa094f-f655-4b75-8a90-f1a2536ce9d6" containerName="istio-proxy" Nov 02 12:37:00.707620 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707217 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="c670745e-5302-4d5d-b3a6-2d5d2b5b00f7" containerName="main" Nov 02 12:37:00.707620 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.707223 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb392c56-19f5-4737-8846-7bf9b0911f1e" containerName="istio-proxy" Nov 02 12:37:00.710134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.710117 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.762250 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.762224 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-02c1d2\"/\"openshift-service-ca.crt\"" Nov 02 12:37:00.799878 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.799847 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-home\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.799999 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.799959 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tmp-dir\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.800049 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.800012 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tls-certs\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.800049 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.800037 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-dshm\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.800128 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.800053 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7tf\" (UniqueName: \"kubernetes.io/projected/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-kube-api-access-4j7tf\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.800128 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.800110 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-model-cache\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.811410 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.811384 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-02c1d2\"/\"kube-root-ca.crt\"" Nov 02 12:37:00.811568 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.811553 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-02c1d2\"/\"noten-model-02c1d2-kserve-self-signed-certs\"" Nov 02 12:37:00.811639 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.811575 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-02c1d2\"/\"default-dockercfg-w5lk7\"" Nov 02 12:37:00.813328 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.813313 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c"] Nov 02 12:37:00.896980 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.896949 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:00.901072 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.901050 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tls-certs\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.901197 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.901095 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-dshm\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.901197 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.901124 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7tf\" (UniqueName: \"kubernetes.io/projected/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-kube-api-access-4j7tf\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.901197 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.901159 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-model-cache\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.901357 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.901223 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-home\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.901357 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.901290 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tmp-dir\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.901705 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.901677 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-model-cache\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.901705 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.901694 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-home\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.901875 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.901728 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tmp-dir\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.903296 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.903276 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-dshm\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.903677 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.903659 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tls-certs\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:00.962434 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:00.962372 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7tf\" (UniqueName: \"kubernetes.io/projected/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-kube-api-access-4j7tf\") pod \"noten-model-02c1d2-kserve-5b9ff776bf-fjx4c\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:01.021522 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:01.021474 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:01.400834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:01.400768 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c"] Nov 02 12:37:01.403477 ip-10-0-141-170 kubenswrapper[4312]: W1102 12:37:01.403452 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f21963e_c74d_4c1a_a3b5_f4e83fa7a042.slice/crio-d2e67bf901466a844bb54b00707f06e65c4259de8bb748e31afefbc119e409b5 WatchSource:0}: Error finding container d2e67bf901466a844bb54b00707f06e65c4259de8bb748e31afefbc119e409b5: Status 404 returned error can't find the container with id d2e67bf901466a844bb54b00707f06e65c4259de8bb748e31afefbc119e409b5 Nov 02 12:37:02.066158 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:02.066122 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" event={"ID":"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042","Type":"ContainerStarted","Data":"160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f"} Nov 02 12:37:02.066569 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:02.066174 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" event={"ID":"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042","Type":"ContainerStarted","Data":"d2e67bf901466a844bb54b00707f06e65c4259de8bb748e31afefbc119e409b5"} Nov 02 12:37:02.183079 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:02.183035 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" podStartSLOduration=2.183018933 podStartE2EDuration="2.183018933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:37:02.182183773 +0000 UTC m=+2439.212420015" watchObservedRunningTime="2026-11-02 12:37:02.183018933 +0000 UTC m=+2439.213255184" Nov 02 12:37:02.910133 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:02.910096 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:03.021721 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:03.021697 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:07.788886 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:07.788849 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:11.021692 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:11.021655 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:11.034193 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:11.034167 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:11.116899 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:11.116874 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:12.702388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:12.702353 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:17.328691 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:17.328657 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:17.571770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:17.571733 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:21.831770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:21.831737 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c"] Nov 02 12:37:21.943855 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:21.943819 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:22.198304 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:22.198273 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:22.445723 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:22.445691 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:22.693077 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:22.693049 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:23.621832 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:23.621799 4312 scope.go:122] "RemoveContainer" containerID="f1c945a0f04a391705d065ee55839cc457031e8ab9c2c18ec8dbad9af2a4034f" Nov 02 12:37:27.383670 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:27.383639 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:27.639243 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:27.639169 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:29.373217 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:37:29.373180 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:37:32.205142 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:32.205095 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:36.850621 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:36.850544 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" podUID="4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" containerName="main" containerID="cri-o://160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f" gracePeriod=45 Nov 02 12:37:37.095207 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.095176 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:37.097406 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.097387 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:37.207715 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.207689 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tmp-dir\") pod \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " Nov 02 12:37:37.207847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.207737 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-kube-api-access-4j7tf\" (UniqueName: \"kubernetes.io/projected/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-kube-api-access-4j7tf\") pod \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " Nov 02 12:37:37.207847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.207797 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-dshm\") pod \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " Nov 02 12:37:37.207847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.207836 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-model-cache\") pod \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " Nov 02 12:37:37.208015 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.207879 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tls-certs\") pod \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " Nov 02 12:37:37.208015 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.207910 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-home\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-home\") pod \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\" (UID: \"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042\") " Nov 02 12:37:37.208015 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.207972 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tmp-dir" pod "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" (UID: "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:37:37.208168 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.208136 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-model-cache" pod "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" (UID: "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:37:37.208223 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.208174 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-home" pod "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" (UID: "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:37:37.208276 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.208258 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:37:37.208335 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.208279 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:37:37.208335 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.208295 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:37:37.210037 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.210007 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-kube-api-access-4j7tf" pod "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" (UID: "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042"). InnerVolumeSpecName "kube-api-access-4j7tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:37:37.210145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.210110 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-dshm" pod "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" (UID: "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:37:37.210145 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.210106 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tls-certs" pod "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" (UID: "4f21963e-c74d-4c1a-a3b5-f4e83fa7a042"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 12:37:37.224252 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.224223 4312 generic.go:356] "Generic (PLEG): container finished" podID="4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" containerID="160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f" exitCode=0 Nov 02 12:37:37.224369 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.224268 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" Nov 02 12:37:37.224369 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.224298 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" event={"ID":"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042","Type":"ContainerDied","Data":"160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f"} Nov 02 12:37:37.224369 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.224341 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c" event={"ID":"4f21963e-c74d-4c1a-a3b5-f4e83fa7a042","Type":"ContainerDied","Data":"d2e67bf901466a844bb54b00707f06e65c4259de8bb748e31afefbc119e409b5"} Nov 02 12:37:37.224369 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.224363 4312 scope.go:122] "RemoveContainer" containerID="160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f" Nov 02 12:37:37.233065 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.233048 4312 scope.go:122] "RemoveContainer" containerID="160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f" Nov 02 12:37:37.233347 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:37:37.233329 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f\": container with ID starting with 160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f not found: ID does not exist" containerID="160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f" Nov 02 12:37:37.233391 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.233357 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f"} err="failed to get container status \"160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f\": rpc error: code = NotFound desc = could not find container \"160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f\": container with ID starting with 160d2a198b4bdf2f9616272e41fdb394803bcad2e0c9eea3d313ef7c35dd8a8f not found: ID does not exist" Nov 02 12:37:37.309387 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.309366 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:37:37.309488 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.309392 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:37:37.309488 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.309402 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j7tf\" (UniqueName: \"kubernetes.io/projected/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042-kube-api-access-4j7tf\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:37:37.387230 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.387208 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c"] Nov 02 12:37:37.391247 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:37.391224 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-02c1d2/noten-model-02c1d2-kserve-5b9ff776bf-fjx4c"] Nov 02 12:37:39.074848 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:39.074814 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" path="/var/lib/kubelet/pods/4f21963e-c74d-4c1a-a3b5-f4e83fa7a042/volumes" Nov 02 12:37:41.988206 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:41.988173 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:46.884693 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:46.884652 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:51.787352 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:51.787313 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:55.818784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:55.818748 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:56.066472 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:56.066445 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:56.673708 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:37:56.673653 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:37:59.371222 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:37:59.371187 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:38:01.498185 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:01.498149 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:04.540094 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:04.540062 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:05.018059 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:05.018029 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:05.969623 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:05.969595 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-extmod-b-3a78b4_payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc_56122628-46a5-4439-bc4a-dbea61accc9c/payload-processing/0.log" Nov 02 12:38:06.380092 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:06.380012 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:06.625693 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:06.625661 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:10.798969 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:10.798933 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:11.047002 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:11.046962 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:11.377648 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:11.377620 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:11.690563 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:11.690533 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:11.937547 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:11.937508 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:12.746895 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:12.746861 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:15.876975 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:15.876941 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:17.510070 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:17.510040 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:17.755014 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:17.754974 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:18.003856 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:18.003815 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:18.554585 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:18.554535 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:18.851869 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:18.851784 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:19.142061 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:19.141983 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:19.655915 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:19.655882 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:20.402936 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:20.402906 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:22.936637 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:22.936604 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:26.714642 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:26.714601 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:27.292152 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:27.292111 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:27.921802 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:27.921772 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:28.171326 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:28.171299 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:29.371952 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:29.371914 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:38:29.372788 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:29.372767 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:32.625834 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:32.625800 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:32.871206 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:32.871176 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:34.250655 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:34.250619 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:34.627847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:34.627762 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:34.901428 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:34.901325 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:39.574838 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:39.574785 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:44.147415 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:44.147374 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-6433487f6bd7: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:44.147415 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:44.147404 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:44.147966 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:44.147483 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials podName:56122628-46a5-4439-bc4a-dbea61accc9c nodeName:}" failed. No retries permitted until 2026-11-02 12:38:44.647459255 +0000 UTC m=+2541.677695513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" (UID: "56122628-46a5-4439-bc4a-dbea61accc9c") : secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:44.484873 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:44.484838 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:44.650656 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:44.650627 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-6433487f6bd7: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:44.650656 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:44.650649 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:44.650883 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:44.650719 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials podName:56122628-46a5-4439-bc4a-dbea61accc9c nodeName:}" failed. No retries permitted until 2026-11-02 12:38:45.650693807 +0000 UTC m=+2542.680930036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" (UID: "56122628-46a5-4439-bc4a-dbea61accc9c") : secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:45.659136 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:45.659095 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-6433487f6bd7: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:45.659136 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:45.659130 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:45.659691 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:45.659203 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials podName:56122628-46a5-4439-bc4a-dbea61accc9c nodeName:}" failed. No retries permitted until 2026-11-02 12:38:47.659183318 +0000 UTC m=+2544.689419561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" (UID: "56122628-46a5-4439-bc4a-dbea61accc9c") : secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:47.675596 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:47.675562 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-6433487f6bd7: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:47.675596 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:47.675588 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:47.676024 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:47.675653 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials podName:56122628-46a5-4439-bc4a-dbea61accc9c nodeName:}" failed. No retries permitted until 2026-11-02 12:38:51.675634668 +0000 UTC m=+2548.705870913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" (UID: "56122628-46a5-4439-bc4a-dbea61accc9c") : secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:49.398275 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:49.398241 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:51.709808 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:51.709769 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-6433487f6bd7: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:51.709808 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:51.709797 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:51.710299 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:51.709874 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials podName:56122628-46a5-4439-bc4a-dbea61accc9c nodeName:}" failed. No retries permitted until 2026-11-02 12:38:59.709860182 +0000 UTC m=+2556.740096410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" (UID: "56122628-46a5-4439-bc4a-dbea61accc9c") : secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:54.300447 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:54.300405 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:56.730415 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:56.730376 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:56.977699 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:56.977666 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:59.139614 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:38:59.139567 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:38:59.373228 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:59.373198 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:38:59.783934 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:59.783905 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-6433487f6bd7: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:59.783934 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:59.783928 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc: secret "agc-cred-6433487f6bd7" not found Nov 02 12:38:59.784135 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:38:59.783981 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials podName:56122628-46a5-4439-bc4a-dbea61accc9c nodeName:}" failed. No retries permitted until 2026-11-02 12:39:15.783965666 +0000 UTC m=+2572.814201896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" (UID: "56122628-46a5-4439-bc4a-dbea61accc9c") : secret "agc-cred-6433487f6bd7" not found Nov 02 12:39:04.076456 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:04.076422 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:08.918099 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:08.918008 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:10.553702 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:10.553669 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:10.799163 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:10.799127 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:13.779007 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:13.778963 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:15.820593 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:39:15.820560 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-4b8a830d91a3: secret "agc-cred-4b8a830d91a3" not found Nov 02 12:39:15.820593 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:39:15.820588 4312 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-3a78b4/agc-cred-6433487f6bd7: secret "agc-cred-6433487f6bd7" not found Nov 02 12:39:15.820593 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:39:15.820597 4312 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc: [secret "agc-cred-4b8a830d91a3" not found, secret "agc-cred-6433487f6bd7" not found] Nov 02 12:39:15.821117 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:39:15.820653 4312 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials podName:56122628-46a5-4439-bc4a-dbea61accc9c nodeName:}" failed. No retries permitted until 2026-11-02 12:39:47.820639654 +0000 UTC m=+2604.850875882 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" (UID: "56122628-46a5-4439-bc4a-dbea61accc9c") : [secret "agc-cred-4b8a830d91a3" not found, secret "agc-cred-6433487f6bd7" not found] Nov 02 12:39:18.144698 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:18.144660 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:18.685386 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:18.685354 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:23.477856 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:23.477820 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:23.723783 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:23.723731 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:28.481271 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:28.481237 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:29.371930 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:39:29.371895 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:39:33.382384 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:33.382341 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:36.216913 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:36.216879 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:36.467015 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:36.466941 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc"] Nov 02 12:39:36.469871 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:36.469844 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:36.723230 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:36.723133 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:37.259799 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:37.259766 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:38.628710 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:38.628672 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:41.467451 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.467395 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" podUID="56122628-46a5-4439-bc4a-dbea61accc9c" containerName="payload-processing" containerID="cri-o://968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6" gracePeriod=25 Nov 02 12:39:41.710813 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.710791 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:39:41.775914 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.775856 4312 generic.go:356] "Generic (PLEG): container finished" podID="56122628-46a5-4439-bc4a-dbea61accc9c" containerID="968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6" exitCode=0 Nov 02 12:39:41.775914 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.775898 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" Nov 02 12:39:41.776059 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.775935 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" event={"ID":"56122628-46a5-4439-bc4a-dbea61accc9c","Type":"ContainerDied","Data":"968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6"} Nov 02 12:39:41.776059 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.775990 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc" event={"ID":"56122628-46a5-4439-bc4a-dbea61accc9c","Type":"ContainerDied","Data":"22800108a05245d6ff38253c4eeff3aec26f6f40f19b4b71bf244c5009a9d511"} Nov 02 12:39:41.776059 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.776013 4312 scope.go:122] "RemoveContainer" containerID="968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6" Nov 02 12:39:41.785436 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.785413 4312 scope.go:122] "RemoveContainer" containerID="968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6" Nov 02 12:39:41.785848 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:39:41.785828 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6\": container with ID starting with 968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6 not found: ID does not exist" containerID="968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6" Nov 02 12:39:41.785924 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.785859 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6"} err="failed to get container status \"968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6\": rpc error: code = NotFound desc = could not find container \"968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6\": container with ID starting with 968b902c8dd5fcd9b1db214102e29faf85a21c92911504f0d5ef183f8d0681f6 not found: ID does not exist" Nov 02 12:39:41.834978 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.834949 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-kube-api-access-xlfpn\" (UniqueName: \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-kube-api-access-xlfpn\") pod \"56122628-46a5-4439-bc4a-dbea61accc9c\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " Nov 02 12:39:41.835079 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.835022 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-routing-overlay-volume\") pod \"56122628-46a5-4439-bc4a-dbea61accc9c\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " Nov 02 12:39:41.835079 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.835065 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-plugins-config-volume\") pod \"56122628-46a5-4439-bc4a-dbea61accc9c\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " Nov 02 12:39:41.835218 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.835200 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials\") pod \"56122628-46a5-4439-bc4a-dbea61accc9c\" (UID: \"56122628-46a5-4439-bc4a-dbea61accc9c\") " Nov 02 12:39:41.835388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.835363 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-routing-overlay-volume" pod "56122628-46a5-4439-bc4a-dbea61accc9c" (UID: "56122628-46a5-4439-bc4a-dbea61accc9c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:39:41.835513 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.835473 4312 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-routing-overlay-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:39:41.835588 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.835529 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-plugins-config-volume" pod "56122628-46a5-4439-bc4a-dbea61accc9c" (UID: "56122628-46a5-4439-bc4a-dbea61accc9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:39:41.837027 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.836999 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-kube-api-access-xlfpn" pod "56122628-46a5-4439-bc4a-dbea61accc9c" (UID: "56122628-46a5-4439-bc4a-dbea61accc9c"). InnerVolumeSpecName "kube-api-access-xlfpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:39:41.837191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.837172 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials" pod "56122628-46a5-4439-bc4a-dbea61accc9c" (UID: "56122628-46a5-4439-bc4a-dbea61accc9c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:39:41.936018 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.935991 4312 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-provider-credentials\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:39:41.936018 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.936014 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlfpn\" (UniqueName: \"kubernetes.io/projected/56122628-46a5-4439-bc4a-dbea61accc9c-kube-api-access-xlfpn\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:39:41.936154 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:41.936024 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56122628-46a5-4439-bc4a-dbea61accc9c-plugins-config-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:39:42.194875 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:42.194846 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc"] Nov 02 12:39:42.199181 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:42.199159 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-3a78b4/payload-processing-external-model-e2e-extmod-b-3a78b4-55ff8gmhc"] Nov 02 12:39:43.075165 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:43.075130 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56122628-46a5-4439-bc4a-dbea61accc9c" path="/var/lib/kubelet/pods/56122628-46a5-4439-bc4a-dbea61accc9c/volumes" Nov 02 12:39:43.526083 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:43.526048 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:43.778589 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:43.778492 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:48.098848 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:48.098813 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:49.859952 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:49.859899 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:50.106352 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:50.106316 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:50.577964 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:50.577924 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:50.998674 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:50.998644 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:52.904905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:52.904874 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:53.828548 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:53.828486 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:57.800210 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:57.800159 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:39:58.029049 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:58.029021 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz"] Nov 02 12:39:58.029285 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:39:58.029263 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" containerID="cri-o://515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed" gracePeriod=30 Nov 02 12:39:59.373725 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:39:59.373683 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:40:00.781047 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:00.781013 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-lvhwz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 12:40:00.781524 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:00.781100 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 12:40:02.701327 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:02.701279 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:03.279513 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.279479 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:40:03.302720 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.302669 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-kube-api-access-6sw9p\" (UniqueName: \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-kube-api-access-6sw9p\") pod \"a83952ed-e3f8-42e2-b779-cec3070a5e02\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " Nov 02 12:40:03.302720 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.302708 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-token\" (UniqueName: \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-token\") pod \"a83952ed-e3f8-42e2-b779-cec3070a5e02\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " Nov 02 12:40:03.302869 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.302735 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-envoy\") pod \"a83952ed-e3f8-42e2-b779-cec3070a5e02\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " Nov 02 12:40:03.302869 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.302761 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-data\") pod \"a83952ed-e3f8-42e2-b779-cec3070a5e02\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " Nov 02 12:40:03.302869 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.302786 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-certs\") pod \"a83952ed-e3f8-42e2-b779-cec3070a5e02\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " Nov 02 12:40:03.302869 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.302817 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83952ed-e3f8-42e2-b779-cec3070a5e02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83952ed-e3f8-42e2-b779-cec3070a5e02-istiod-ca-cert\") pod \"a83952ed-e3f8-42e2-b779-cec3070a5e02\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " Nov 02 12:40:03.303085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.302885 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-credential-socket\") pod \"a83952ed-e3f8-42e2-b779-cec3070a5e02\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " Nov 02 12:40:03.303085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.302919 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-podinfo\") pod \"a83952ed-e3f8-42e2-b779-cec3070a5e02\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " Nov 02 12:40:03.303085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.302945 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-socket\") pod \"a83952ed-e3f8-42e2-b779-cec3070a5e02\" (UID: \"a83952ed-e3f8-42e2-b779-cec3070a5e02\") " Nov 02 12:40:03.303085 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.303043 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-certs" pod "a83952ed-e3f8-42e2-b779-cec3070a5e02" (UID: "a83952ed-e3f8-42e2-b779-cec3070a5e02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:40:03.303301 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.303076 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-data" pod "a83952ed-e3f8-42e2-b779-cec3070a5e02" (UID: "a83952ed-e3f8-42e2-b779-cec3070a5e02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:40:03.303301 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.303226 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-socket" pod "a83952ed-e3f8-42e2-b779-cec3070a5e02" (UID: "a83952ed-e3f8-42e2-b779-cec3070a5e02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:40:03.303301 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.303240 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-credential-socket" pod "a83952ed-e3f8-42e2-b779-cec3070a5e02" (UID: "a83952ed-e3f8-42e2-b779-cec3070a5e02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:40:03.303422 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.303297 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:40:03.303422 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.303309 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83952ed-e3f8-42e2-b779-cec3070a5e02-istiod-ca-cert" pod "a83952ed-e3f8-42e2-b779-cec3070a5e02" (UID: "a83952ed-e3f8-42e2-b779-cec3070a5e02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:40:03.303422 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.303318 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:40:03.304766 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.304744 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-kube-api-access-6sw9p" pod "a83952ed-e3f8-42e2-b779-cec3070a5e02" (UID: "a83952ed-e3f8-42e2-b779-cec3070a5e02"). InnerVolumeSpecName "kube-api-access-6sw9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:40:03.305283 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.305262 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-envoy" pod "a83952ed-e3f8-42e2-b779-cec3070a5e02" (UID: "a83952ed-e3f8-42e2-b779-cec3070a5e02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:40:03.305364 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.305289 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-token" pod "a83952ed-e3f8-42e2-b779-cec3070a5e02" (UID: "a83952ed-e3f8-42e2-b779-cec3070a5e02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:40:03.305364 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.305340 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-podinfo" pod "a83952ed-e3f8-42e2-b779-cec3070a5e02" (UID: "a83952ed-e3f8-42e2-b779-cec3070a5e02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:40:03.404629 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.404604 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:40:03.404629 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.404625 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:40:03.404757 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.404635 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sw9p\" (UniqueName: \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-kube-api-access-6sw9p\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:40:03.404757 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.404645 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:40:03.404757 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.404653 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:40:03.404757 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.404661 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a83952ed-e3f8-42e2-b779-cec3070a5e02-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:40:03.404757 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.404670 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a83952ed-e3f8-42e2-b779-cec3070a5e02-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:40:03.880733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.880692 4312 generic.go:356] "Generic (PLEG): container finished" podID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerID="515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed" exitCode=0 Nov 02 12:40:03.881184 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.880758 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" Nov 02 12:40:03.881184 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.880755 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" event={"ID":"a83952ed-e3f8-42e2-b779-cec3070a5e02","Type":"ContainerDied","Data":"515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed"} Nov 02 12:40:03.881184 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.880889 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz" event={"ID":"a83952ed-e3f8-42e2-b779-cec3070a5e02","Type":"ContainerDied","Data":"587aeb86268070634f3fa77b1adb99859d8382077e1121bc435f69acec47ff1f"} Nov 02 12:40:03.881184 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.880919 4312 scope.go:122] "RemoveContainer" containerID="515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed" Nov 02 12:40:03.891986 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.891966 4312 scope.go:122] "RemoveContainer" containerID="515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed" Nov 02 12:40:03.892282 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:40:03.892258 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed\": container with ID starting with 515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed not found: ID does not exist" containerID="515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed" Nov 02 12:40:03.892322 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:03.892292 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed"} err="failed to get container status \"515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed\": rpc error: code = NotFound desc = could not find container \"515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed\": container with ID starting with 515976d0ddab90e638bd5ed41dbe1dd11f6906b072d194c6f9cd6f33dbfd27ed not found: ID does not exist" Nov 02 12:40:04.044874 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:04.044849 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz"] Nov 02 12:40:04.094376 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:04.094348 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-lvhwz"] Nov 02 12:40:05.074474 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:05.074434 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" path="/var/lib/kubelet/pods/a83952ed-e3f8-42e2-b779-cec3070a5e02/volumes" Nov 02 12:40:07.596342 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:07.596305 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:12.409113 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:12.409077 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:17.296020 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:17.295989 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:18.487792 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:18.487747 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:22.186987 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:22.186939 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:27.075662 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:27.075630 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:29.372269 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:40:29.372229 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:40:32.005759 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:32.005727 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:36.923838 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:36.923753 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:41.804915 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:41.804878 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:43.303734 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:43.303698 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:43.557222 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:43.557134 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:43.806463 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:43.806430 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:44.274667 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:44.274634 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:44.534666 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:44.534590 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:44.783535 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:44.783480 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:45.032257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:45.032225 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:45.807192 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:45.807159 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:46.053538 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:46.053484 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:46.792520 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:46.792470 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:48.413453 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:48.413420 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:49.225805 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:49.225769 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:49.562919 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:49.562839 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:50.037809 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:50.037775 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:51.753315 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:51.753288 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:56.537463 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:40:56.537418 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:40:59.375245 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:40:59.375203 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:41:01.406988 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:01.406956 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:06.282433 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:06.282399 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:11.193165 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:11.193120 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:16.096899 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:16.096858 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:18.833316 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:18.833278 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:20.988889 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:20.988853 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:25.890388 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:25.890353 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:29.371710 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:41:29.371674 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:41:30.794287 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:30.794259 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:35.599452 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:35.599403 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:40.580556 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:40.580515 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:44.929380 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:44.929343 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:45.180385 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:45.180275 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:45.526299 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:45.526224 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:46.376694 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:46.376660 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:46.625619 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:46.625578 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:50.392883 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:50.392844 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:55.342118 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:41:55.342082 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:41:59.373228 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:41:59.373190 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:42:00.191794 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:00.191742 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:05.088061 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:05.088024 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:09.979914 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:09.979817 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:14.889091 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:14.889025 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:19.180415 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:19.180378 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:19.789912 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:19.789879 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:24.601103 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:24.601065 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:29.373032 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:42:29.372994 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:42:29.484658 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:29.484615 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:34.298857 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:34.298818 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:39.186154 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:39.186120 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:43.936710 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:43.936666 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:44.188143 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:44.188054 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:48.984546 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:48.984511 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:53.883738 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:53.883702 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:58.800530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:58.800474 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:59.045174 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:42:59.045139 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:42:59.372288 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:42:59.372253 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:43:02.476029 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:02.475998 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:02.726199 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:02.726129 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:03.676718 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:03.676683 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:08.500586 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:08.500541 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:13.392191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:13.392153 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:14.843103 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:14.843067 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:18.288563 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:18.288531 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:19.525789 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:19.525748 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:23.186918 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:23.186886 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:28.115307 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:28.115275 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:29.371942 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:43:29.371907 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:43:32.987693 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:32.987660 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:37.898272 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:37.898181 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:42.789871 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:42.789829 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:44.769296 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:44.769262 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng"] Nov 02 12:43:44.769830 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:44.769486 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" containerID="cri-o://e036f8245438477274dc795f8ddd9dabd9c894d274319656932da9665653c715" gracePeriod=30 Nov 02 12:43:47.696605 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:47.696570 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:49.896904 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:49.896876 4312 generic.go:356] "Generic (PLEG): container finished" podID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerID="e036f8245438477274dc795f8ddd9dabd9c894d274319656932da9665653c715" exitCode=0 Nov 02 12:43:49.897208 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:49.896925 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" event={"ID":"d7f4312e-6933-473d-ace6-29cb4b45512d","Type":"ContainerDied","Data":"e036f8245438477274dc795f8ddd9dabd9c894d274319656932da9665653c715"} Nov 02 12:43:50.019222 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.019193 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:43:50.085573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085515 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-data\") pod \"d7f4312e-6933-473d-ace6-29cb4b45512d\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " Nov 02 12:43:50.085573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085554 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-credential-socket\") pod \"d7f4312e-6933-473d-ace6-29cb4b45512d\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " Nov 02 12:43:50.085744 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085581 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f4312e-6933-473d-ace6-29cb4b45512d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f4312e-6933-473d-ace6-29cb4b45512d-istiod-ca-cert\") pod \"d7f4312e-6933-473d-ace6-29cb4b45512d\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " Nov 02 12:43:50.085744 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085604 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-certs\") pod \"d7f4312e-6933-473d-ace6-29cb4b45512d\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " Nov 02 12:43:50.085744 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085634 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-envoy\") pod \"d7f4312e-6933-473d-ace6-29cb4b45512d\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " Nov 02 12:43:50.085744 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085663 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-token\" (UniqueName: \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-token\") pod \"d7f4312e-6933-473d-ace6-29cb4b45512d\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " Nov 02 12:43:50.085744 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085691 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-kube-api-access-b8dgw\" (UniqueName: \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-kube-api-access-b8dgw\") pod \"d7f4312e-6933-473d-ace6-29cb4b45512d\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " Nov 02 12:43:50.085744 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085730 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-socket\") pod \"d7f4312e-6933-473d-ace6-29cb4b45512d\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " Nov 02 12:43:50.086042 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085768 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-data" pod "d7f4312e-6933-473d-ace6-29cb4b45512d" (UID: "d7f4312e-6933-473d-ace6-29cb4b45512d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:43:50.086042 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085796 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-podinfo\") pod \"d7f4312e-6933-473d-ace6-29cb4b45512d\" (UID: \"d7f4312e-6933-473d-ace6-29cb4b45512d\") " Nov 02 12:43:50.086042 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.086008 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:43:50.086042 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085793 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-credential-socket" pod "d7f4312e-6933-473d-ace6-29cb4b45512d" (UID: "d7f4312e-6933-473d-ace6-29cb4b45512d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:43:50.086042 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.085880 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-certs" pod "d7f4312e-6933-473d-ace6-29cb4b45512d" (UID: "d7f4312e-6933-473d-ace6-29cb4b45512d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:43:50.086259 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.086040 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-socket" pod "d7f4312e-6933-473d-ace6-29cb4b45512d" (UID: "d7f4312e-6933-473d-ace6-29cb4b45512d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:43:50.086259 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.086196 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f4312e-6933-473d-ace6-29cb4b45512d-istiod-ca-cert" pod "d7f4312e-6933-473d-ace6-29cb4b45512d" (UID: "d7f4312e-6933-473d-ace6-29cb4b45512d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 12:43:50.087863 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.087837 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-podinfo" pod "d7f4312e-6933-473d-ace6-29cb4b45512d" (UID: "d7f4312e-6933-473d-ace6-29cb4b45512d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 12:43:50.088013 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.087948 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-kube-api-access-b8dgw" pod "d7f4312e-6933-473d-ace6-29cb4b45512d" (UID: "d7f4312e-6933-473d-ace6-29cb4b45512d"). InnerVolumeSpecName "kube-api-access-b8dgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:43:50.088292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.088268 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-token" pod "d7f4312e-6933-473d-ace6-29cb4b45512d" (UID: "d7f4312e-6933-473d-ace6-29cb4b45512d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 12:43:50.088397 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.088275 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-envoy" pod "d7f4312e-6933-473d-ace6-29cb4b45512d" (UID: "d7f4312e-6933-473d-ace6-29cb4b45512d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 12:43:50.187310 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.187285 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:43:50.187449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.187314 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:43:50.187449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.187330 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:43:50.187449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.187343 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f4312e-6933-473d-ace6-29cb4b45512d-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:43:50.187449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.187358 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:43:50.187449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.187372 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:43:50.187449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.187386 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:43:50.187449 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.187410 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8dgw\" (UniqueName: \"kubernetes.io/projected/d7f4312e-6933-473d-ace6-29cb4b45512d-kube-api-access-b8dgw\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 12:43:50.903231 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.903197 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" Nov 02 12:43:50.903796 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.903228 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng" event={"ID":"d7f4312e-6933-473d-ace6-29cb4b45512d","Type":"ContainerDied","Data":"5ff6be6375be9b8f29fb475d0a65d9dc397241b7f7402dad97072bdb0d7143f5"} Nov 02 12:43:50.903796 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:50.903308 4312 scope.go:122] "RemoveContainer" containerID="e036f8245438477274dc795f8ddd9dabd9c894d274319656932da9665653c715" Nov 02 12:43:51.119626 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:51.119596 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng"] Nov 02 12:43:51.125817 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:51.125795 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-th4ng"] Nov 02 12:43:52.596471 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:52.596417 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:53.074610 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:53.074581 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" path="/var/lib/kubelet/pods/d7f4312e-6933-473d-ace6-29cb4b45512d/volumes" Nov 02 12:43:57.494289 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:43:57.494212 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:43:59.373017 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:43:59.372978 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:44:02.422051 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:02.421991 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:07.280043 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:07.280003 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:12.198974 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:12.198933 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:17.100935 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:17.100850 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:19.862302 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:19.862256 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:22.002454 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:22.002379 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:26.908828 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:26.908775 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:29.373444 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:44:29.373399 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:44:31.801605 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:31.801569 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:34.340645 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:34.340609 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:34.587766 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:34.587735 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:36.695144 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:36.695088 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:38.296532 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:38.296484 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:38.546647 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:38.546542 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:41.599846 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:41.599797 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:46.483577 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:46.483536 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:51.382046 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:51.382009 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:56.287161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:44:56.287121 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:44:59.371912 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:44:59.371874 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:45:01.190063 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:01.190012 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:06.098292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:06.098213 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:10.995019 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:10.994984 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:15.885573 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:15.885535 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:20.214939 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:20.214910 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:20.463344 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:20.463315 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:20.811967 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:20.811921 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:25.693663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:25.693574 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:29.374374 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:45:29.374337 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:45:30.590971 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:30.590923 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:35.496005 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:35.495966 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:40.395010 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:40.394972 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:45.278733 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:45.278703 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:50.204967 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:50.204934 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:55.115317 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:55.115259 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:45:59.374042 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:45:59.374001 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:45:59.982021 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:45:59.981952 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:04.798105 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:04.798064 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:09.691478 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:09.691446 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:14.506293 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:14.506252 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:19.319785 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:19.319706 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:20.553761 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:20.553698 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:24.198383 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:24.198337 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:29.088182 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:29.088126 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:29.371732 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:46:29.371654 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:46:31.004403 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:31.004355 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:31.260701 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:31.260603 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:33.922753 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:33.922697 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:38.801965 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:38.801861 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:43.736201 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:43.736156 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:48.587964 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:48.587921 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:53.489716 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:53.489683 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:58.435570 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:46:58.435528 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:46:59.375643 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:46:59.375603 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:47:03.279209 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:03.279171 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:08.191355 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:08.191323 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:13.089025 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:13.088984 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:17.917558 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:17.917522 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:20.905068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:20.905031 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:22.797195 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:22.797145 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:27.685750 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:27.685718 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:29.373249 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:47:29.373210 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:47:32.578483 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:32.578442 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:37.492150 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:37.492112 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:42.385247 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:42.385211 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:47.288457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:47.288423 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:52.214217 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:52.214180 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:57.079013 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:47:57.078977 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:47:59.374178 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:47:59.374138 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:48:02.070672 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:02.070634 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:06.895464 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:06.895380 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:11.792131 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:11.792092 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:16.694686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:16.694644 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:21.242583 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:21.242543 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:21.490068 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:21.490031 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:21.734213 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:21.734180 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:26.511106 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:26.511066 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:29.373804 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:48:29.373764 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:48:31.386597 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:31.386558 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:36.287534 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:36.287483 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:41.178284 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:41.178249 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:46.025799 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:46.025764 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:50.926005 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:50.925969 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:55.784574 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:48:55.784528 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:48:59.372753 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:48:59.372718 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:49:00.682925 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:00.682889 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:05.583143 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:05.583106 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:08.292649 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:08.292614 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:08.541074 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:08.541038 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:10.484784 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:10.484743 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:11.826063 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:11.826031 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:12.074609 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:12.074581 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:15.417297 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:15.417263 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:20.383686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:20.383649 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:21.596309 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:21.596272 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:25.283813 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:25.283774 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:29.372523 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:49:29.372471 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:49:30.090871 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:30.090838 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:35.003957 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:35.003917 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:39.889585 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:39.889455 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:46.724034 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:46.724001 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:49.770864 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:49.770827 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:54.589931 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:54.589895 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:49:59.372229 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:49:59.372195 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:49:59.479427 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:49:59.479397 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:04.391893 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:04.391858 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:09.282111 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:09.282080 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:14.187908 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:14.187875 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:19.083539 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:19.083485 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:21.934070 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:21.934037 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:23.989225 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:23.989188 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:28.882477 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:28.882443 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:29.371927 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:50:29.371893 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:50:33.783910 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:33.783869 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:38.620291 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:38.620254 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:43.482457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:43.482420 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:48.387663 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:48.387627 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:53.206822 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:53.206789 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:57.092564 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:57.092519 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:58.100871 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:50:58.100837 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:50:59.372440 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:50:59.372404 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:51:02.990237 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:02.990200 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:07.888367 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:07.888282 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:12.801592 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:12.801552 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:17.692234 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:17.692197 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:22.284673 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:22.284633 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:22.529845 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:22.529813 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:22.776650 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:22.776621 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:27.482577 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:27.482543 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:29.372915 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:51:29.372883 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:51:32.385556 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:32.385518 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:37.199102 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:37.199058 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:41.998319 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:41.998283 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:46.994227 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:46.994191 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:51.495788 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:51.495759 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:51.753750 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:51.753678 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:52.004253 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:52.004173 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:56.786917 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:51:56.786887 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:51:59.373714 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:51:59.373671 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:52:01.751696 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:01.751617 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:06.588929 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:06.588899 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:06.859722 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:06.859652 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:07.109123 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:07.109094 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:11.487268 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:11.487230 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:15.797739 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:15.797708 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:16.389299 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:16.389265 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:21.283092 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:21.283034 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:22.630654 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:22.630613 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:26.111013 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:26.110980 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:29.372149 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:52:29.372111 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:52:31.098373 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:31.098322 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:35.991338 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:35.991234 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:40.876198 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:40.876134 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:45.782847 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:45.782804 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:50.701855 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:50.701821 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:55.583905 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:52:55.583861 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:52:59.372086 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:52:59.372050 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:53:00.479880 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:00.479841 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:02.076257 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:02.076221 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:05.296686 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:05.296644 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:10.187976 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:10.187928 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:15.095550 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:15.095469 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:19.989925 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:19.989883 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:22.963060 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:22.963021 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:24.980346 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:24.980305 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:29.374033 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:53:29.373993 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:53:29.908368 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:29.908325 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:34.774666 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:34.774631 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:39.597579 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:39.597534 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:44.497991 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:44.497941 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:49.428323 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:49.428287 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:54.293911 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:54.293880 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:59.201701 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:53:59.201663 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:53:59.372227 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:53:59.372196 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:54:04.092341 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:04.092308 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:08.986268 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:08.986180 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:13.902661 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:13.902618 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:18.793265 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:18.793228 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:23.312964 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:23.312933 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:23.560612 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:23.560579 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:23.898517 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:23.898469 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:28.795895 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:28.795856 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:29.374023 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:54:29.373981 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:54:33.795244 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:33.795205 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:36.342103 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:36.342068 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:36.597901 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:36.597820 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:38.701900 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:38.701865 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:43.594295 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:43.594259 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:48.414906 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:48.414865 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:53.278479 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:53.278442 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:58.185819 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:54:58.185782 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:54:59.376009 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:54:59.375957 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:55:03.075903 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:03.075869 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:07.892253 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:07.892213 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:12.794713 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:12.794684 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:17.686161 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:17.686126 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:22.599854 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:22.599810 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:23.652292 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:23.652251 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:27.481614 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:27.481577 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:29.375327 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:55:29.375283 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:55:32.373277 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:32.373239 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:37.202648 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:37.202547 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:42.095516 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:42.095463 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:46.903468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:46.903436 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:51.788656 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:51.788616 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:56.686814 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:55:56.686772 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:55:59.374624 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:55:59.374586 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:56:01.612590 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:01.612554 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:06.413301 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:06.413253 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:11.292875 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:11.292839 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:16.198389 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:16.198337 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:21.093378 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:21.093330 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:22.214351 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:22.214317 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:24.014330 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:24.014287 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:25.994425 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:25.994392 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:29.373262 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:56:29.373219 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:56:30.880154 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:30.880115 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:35.800344 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:35.800301 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:40.699567 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:40.699525 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:45.503380 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:45.503346 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:50.493330 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:50.493281 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:55.394803 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:56:55.394766 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:56:59.373102 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:56:59.373065 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:57:00.299233 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:00.299196 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:03.387939 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:03.387893 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:05.199159 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:05.199125 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:10.076191 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:10.076154 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:14.909548 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:14.909514 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:19.780650 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:19.780609 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:24.341652 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:24.341615 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:24.687134 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:24.687105 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:29.373763 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:57:29.373727 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:57:29.582046 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:29.582016 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:34.474072 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:34.474036 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:39.392008 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:39.391971 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:44.289840 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:44.289808 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:49.187916 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:49.187883 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:54.091530 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:54.091484 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:58.997078 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:57:58.997034 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:57:59.374014 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:57:59.373925 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:58:03.892925 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:03.892875 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:08.798417 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:08.798374 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:13.692443 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:13.692403 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:18.598804 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:18.598759 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:23.490250 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:23.490218 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:24.688425 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:24.688389 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:28.411294 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:28.411253 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:29.373208 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:58:29.373174 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:58:33.297192 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:33.297156 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:38.195001 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:38.194914 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:42.209132 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:42.209102 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:42.458790 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:42.458756 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:43.107000 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:43.106971 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:47.984985 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:47.984949 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:52.887021 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:52.886980 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:57.700440 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:58:57.700405 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:58:59.371740 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:58:59.371704 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:59:02.590293 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:02.590260 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:06.836619 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:06.836584 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:07.085223 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:07.085196 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:07.498390 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:07.498353 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:09.297120 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:09.297084 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:09.546250 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:09.546216 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:12.396668 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:12.396629 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:17.285774 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:17.285742 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:19.718457 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:19.718421 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:22.190775 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:22.190744 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:24.827938 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:24.827901 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:25.072364 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:25.072328 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:26.196167 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:26.196133 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:26.443411 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:26.443379 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:26.806179 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:26.806137 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:27.051135 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:27.051103 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:27.299096 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:27.299070 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:28.012684 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:28.012652 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:28.257468 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:28.257432 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:28.771303 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:28.771268 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:29.372239 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:59:29.372206 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 12:59:31.998775 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:31.998729 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:34.765124 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:34.765086 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:35.011770 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:35.011735 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:36.901772 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:36.901724 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:40.856556 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:40.856515 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:41.104851 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:41.104819 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:41.792677 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:41.792638 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:42.090337 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:42.090265 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:42.334786 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:42.334755 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:46.228321 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:46.228287 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:46.786148 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:46.786118 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:51.697208 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:51.697173 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:56.536689 ip-10-0-141-170 kubenswrapper[4312]: I1102 12:59:56.536654 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 12:59:59.373118 ip-10-0-141-170 kubenswrapper[4312]: E1102 12:59:59.373080 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:00:01.388761 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:01.388727 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:06.298445 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:06.298355 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:11.199431 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:11.199391 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:16.092360 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:16.092312 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:19.174849 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.174816 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955"] Nov 02 13:00:19.175247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175132 4312 cpu_manager.go:414] "containerMap: removing container" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" Nov 02 13:00:19.175247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175143 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" Nov 02 13:00:19.175247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175155 4312 cpu_manager.go:414] "containerMap: removing container" podUID="4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" containerName="main" Nov 02 13:00:19.175247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175160 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" containerName="main" Nov 02 13:00:19.175247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175168 4312 cpu_manager.go:414] "containerMap: removing container" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" Nov 02 13:00:19.175247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175172 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" Nov 02 13:00:19.175247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175184 4312 cpu_manager.go:414] "containerMap: removing container" podUID="56122628-46a5-4439-bc4a-dbea61accc9c" containerName="payload-processing" Nov 02 13:00:19.175247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175189 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56122628-46a5-4439-bc4a-dbea61accc9c" containerName="payload-processing" Nov 02 13:00:19.175559 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175260 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f4312e-6933-473d-ace6-29cb4b45512d" containerName="istio-proxy" Nov 02 13:00:19.175559 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175267 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f21963e-c74d-4c1a-a3b5-f4e83fa7a042" containerName="main" Nov 02 13:00:19.175559 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175274 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83952ed-e3f8-42e2-b779-cec3070a5e02" containerName="istio-proxy" Nov 02 13:00:19.175559 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.175281 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="56122628-46a5-4439-bc4a-dbea61accc9c" containerName="payload-processing" Nov 02 13:00:19.178270 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.178251 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.232250 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.232228 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 13:00:19.282541 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.282514 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955"] Nov 02 13:00:19.307139 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.307110 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.307249 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.307146 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.307249 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.307183 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.307324 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.307272 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.307324 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.307306 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.307395 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.307331 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.307395 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.307346 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.307459 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.307419 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.307515 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.307478 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs9z7\" (UniqueName: \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-kube-api-access-fs9z7\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.330639 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.330616 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c5ldd\"" Nov 02 13:00:19.408471 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408441 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.408614 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408480 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.408614 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408538 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.408614 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408562 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.408614 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408584 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.408843 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408624 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.408843 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408661 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs9z7\" (UniqueName: \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-kube-api-access-fs9z7\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.408843 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408715 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.408843 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408749 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.409062 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408884 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.409062 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408944 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.409062 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.408964 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.409292 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.409269 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.409360 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.409294 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.410960 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.410942 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.411115 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.411099 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.469817 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.469767 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.469929 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.469912 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs9z7\" (UniqueName: \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-kube-api-access-fs9z7\") pod \"maas-default-gateway-openshift-default-6dcf757899-kj955\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.490699 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.490680 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:19.913792 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.913768 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955"] Nov 02 13:00:19.915648 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:00:19.915617 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea83810_a6ec_4e8a_b4b2_00457e869a8f.slice/crio-2cf51ce3edbe4927177f5f8ae95e1c236bbf45f2dcb8cef2aaa51ccdbaa1cdf5 WatchSource:0}: Error finding container 2cf51ce3edbe4927177f5f8ae95e1c236bbf45f2dcb8cef2aaa51ccdbaa1cdf5: Status 404 returned error can't find the container with id 2cf51ce3edbe4927177f5f8ae95e1c236bbf45f2dcb8cef2aaa51ccdbaa1cdf5 Nov 02 13:00:19.917916 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.917876 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:00:19.917991 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.917958 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:00:19.918043 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:19.918023 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:00:20.280606 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:20.280514 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" event={"ID":"8ea83810-a6ec-4e8a-b4b2-00457e869a8f","Type":"ContainerStarted","Data":"2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c"} Nov 02 13:00:20.280606 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:20.280576 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" event={"ID":"8ea83810-a6ec-4e8a-b4b2-00457e869a8f","Type":"ContainerStarted","Data":"2cf51ce3edbe4927177f5f8ae95e1c236bbf45f2dcb8cef2aaa51ccdbaa1cdf5"} Nov 02 13:00:20.402911 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:20.402847 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" podStartSLOduration=1.402831154 podStartE2EDuration="1.402831154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:20.399818423 +0000 UTC m=+3837.430054674" watchObservedRunningTime="2026-11-02 13:00:20.402831154 +0000 UTC m=+3837.433067403" Nov 02 13:00:20.491687 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:20.491651 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:20.493134 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:20.493108 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kj955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 13:00:20.493278 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:20.493163 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 13:00:21.115972 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:21.115944 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:21.491319 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:21.491286 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kj955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 13:00:21.491725 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:21.491333 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 13:00:22.491578 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:22.491542 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kj955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 13:00:22.492061 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:22.491597 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 13:00:23.491670 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:23.491637 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kj955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 13:00:23.492018 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:23.491685 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 13:00:24.491312 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:24.491276 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kj955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 13:00:24.491488 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:24.491329 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 13:00:25.395060 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:25.395027 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:25.491488 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:25.491455 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kj955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 13:00:25.491659 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:25.491528 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 13:00:26.083369 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:26.083341 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:26.491417 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:26.491382 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kj955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 13:00:26.491879 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:26.491433 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 13:00:27.491190 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:27.491151 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-kj955 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 02 13:00:27.491421 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:27.491210 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 02 13:00:29.208541 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:29.208508 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:29.209041 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:29.209013 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:29.209767 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:29.209747 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:00:29.371740 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:00:29.371711 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:00:30.577241 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:30.577206 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:30.826525 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:30.826480 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:31.076421 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:31.076395 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:33.090786 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:33.090741 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:33.339805 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:33.339773 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:35.996176 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:35.996137 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:36.963417 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:36.963381 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:39.387329 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:39.387296 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:39.634399 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:39.634370 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:41.015430 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:41.015398 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:41.264542 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:41.264481 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:41.509700 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:41.509670 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:42.504130 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:42.504094 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:45.563283 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:45.563252 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:45.833786 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:45.833675 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:46.086983 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:46.086910 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:47.428665 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:47.428630 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:50.804429 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:50.804396 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:52.218031 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:52.218001 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:52.465831 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:52.465800 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:54.416890 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:54.416856 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:54.668657 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:54.668586 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:55.690724 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:55.690670 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:58.593941 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:00:58.593906 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:00:59.371725 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:00:59.371691 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:01:00.587371 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:00.587342 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:01.024128 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:01.024094 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:01.276264 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:01.276186 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:02.738547 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:02.738511 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:03.914308 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:03.914271 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:04.184632 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.184535 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:04.444891 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.444817 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:04.521359 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.521326 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5"] Nov 02 13:01:04.524641 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.524625 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.570837 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.570813 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw"] Nov 02 13:01:04.573968 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.573948 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.604243 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.604225 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.604329 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.604258 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.604329 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.604289 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.604434 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.604415 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.604474 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.604451 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.604474 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.604468 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.604577 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.604565 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.604614 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.604591 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4skq8\" (UniqueName: \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-kube-api-access-4skq8\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.604657 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.604644 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.630600 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.630577 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5"] Nov 02 13:01:04.683246 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.683221 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw"] Nov 02 13:01:04.705900 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.705843 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.705900 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.705881 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.705900 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.705900 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.706150 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706136 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.706217 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706158 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.706217 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706176 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.706330 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706249 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.706330 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706278 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4skq8\" (UniqueName: \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-kube-api-access-4skq8\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.706330 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706311 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.706487 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706334 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.706487 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706330 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.706487 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706364 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.706487 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706421 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.706487 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706458 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.706783 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706485 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.706783 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706557 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvtpn\" (UniqueName: \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-kube-api-access-hvtpn\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.706783 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706590 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.706783 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706622 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.706783 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706650 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.706783 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706731 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.707046 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706781 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.707046 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.706904 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.707371 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.707348 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.708578 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.708550 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:04.708578 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.708566 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.708894 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.708878 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.766545 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.766523 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.766773 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.766751 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4skq8\" (UniqueName: \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-kube-api-access-4skq8\") pod \"maas-default-gateway-openshift-default-6dcf757899-r64l5\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.807706 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.807682 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.807819 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.807710 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.807819 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.807729 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.807819 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.807747 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.807819 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.807770 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvtpn\" (UniqueName: \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-kube-api-access-hvtpn\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.807819 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.807794 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.808109 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.807835 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.808109 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.807888 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.808109 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.807929 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.808241 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.808106 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.808241 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.808198 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.808241 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.808212 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.808391 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.808279 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.808662 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.808646 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.810079 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.810059 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.810310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.810287 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.837156 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.837128 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:04.868508 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.868459 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.868508 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.868488 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvtpn\" (UniqueName: \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-kube-api-access-hvtpn\") pod \"maas-default-gateway-openshift-default-6dcf757899-ffnfw\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:04.886318 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:04.886291 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:05.260155 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.260127 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5"] Nov 02 13:01:05.262850 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:01:05.262822 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d962c8_a8b5_48e0_a6af_4162f5932d1d.slice/crio-715795f459a3e15c557927b1b08ecce4fd19742d890cbd51ecfb29cba2ce9a91 WatchSource:0}: Error finding container 715795f459a3e15c557927b1b08ecce4fd19742d890cbd51ecfb29cba2ce9a91: Status 404 returned error can't find the container with id 715795f459a3e15c557927b1b08ecce4fd19742d890cbd51ecfb29cba2ce9a91 Nov 02 13:01:05.265387 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.265354 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:01:05.265516 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.265430 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:01:05.265516 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.265487 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:01:05.316425 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.316403 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw"] Nov 02 13:01:05.322488 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:01:05.322457 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ad9289_bd34_4176_a6be_1f0b348ee3e1.slice/crio-1654c20fcac0be69354d282628cde882f4316b61968695cc0285f1dadceb25a5 WatchSource:0}: Error finding container 1654c20fcac0be69354d282628cde882f4316b61968695cc0285f1dadceb25a5: Status 404 returned error can't find the container with id 1654c20fcac0be69354d282628cde882f4316b61968695cc0285f1dadceb25a5 Nov 02 13:01:05.325277 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.325246 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:01:05.325389 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.325306 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:01:05.325389 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.325335 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:01:05.482479 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.481964 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" event={"ID":"63ad9289-bd34-4176-a6be-1f0b348ee3e1","Type":"ContainerStarted","Data":"de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404"} Nov 02 13:01:05.482479 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.482046 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" event={"ID":"63ad9289-bd34-4176-a6be-1f0b348ee3e1","Type":"ContainerStarted","Data":"1654c20fcac0be69354d282628cde882f4316b61968695cc0285f1dadceb25a5"} Nov 02 13:01:05.483418 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.483351 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:05.484661 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.484638 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" event={"ID":"85d962c8-a8b5-48e0-a6af-4162f5932d1d","Type":"ContainerStarted","Data":"ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1"} Nov 02 13:01:05.484767 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.484682 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" event={"ID":"85d962c8-a8b5-48e0-a6af-4162f5932d1d","Type":"ContainerStarted","Data":"715795f459a3e15c557927b1b08ecce4fd19742d890cbd51ecfb29cba2ce9a91"} Nov 02 13:01:05.603901 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.603796 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" podStartSLOduration=1.6037783110000001 podStartE2EDuration="1.603778311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:05.602225365 +0000 UTC m=+3882.632461615" watchObservedRunningTime="2026-11-02 13:01:05.603778311 +0000 UTC m=+3882.634014565" Nov 02 13:01:05.725110 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.725057 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" podStartSLOduration=1.725040032 podStartE2EDuration="1.725040032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:05.723524351 +0000 UTC m=+3882.753760598" watchObservedRunningTime="2026-11-02 13:01:05.725040032 +0000 UTC m=+3882.755276282" Nov 02 13:01:05.740028 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.739996 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:05.837752 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.837715 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:05.898598 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:05.898125 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:06.839842 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:06.839564 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r64l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 13:01:06.839842 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:06.839624 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 13:01:06.901602 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:06.901561 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ffnfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 13:01:06.901774 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:06.901628 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 13:01:07.838073 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:07.838041 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r64l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 13:01:07.838271 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:07.838106 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 13:01:07.887890 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:07.887854 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ffnfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 13:01:07.888465 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:07.888433 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 13:01:08.838406 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:08.838364 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r64l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 13:01:08.838648 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:08.838435 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 13:01:08.887900 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:08.887866 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ffnfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 13:01:08.888486 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:08.888447 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 13:01:09.189342 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:09.189254 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:09.837867 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:09.837833 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r64l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 13:01:09.838061 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:09.837889 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 13:01:09.888521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:09.887903 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ffnfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 13:01:09.888521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:09.887979 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 13:01:10.528144 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:10.528114 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:10.837881 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:10.837788 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r64l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 13:01:10.837881 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:10.837848 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 13:01:10.893521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:10.889638 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ffnfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 13:01:10.893521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:10.889694 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 13:01:11.838156 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:11.838119 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r64l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 13:01:11.838331 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:11.838171 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 13:01:11.887692 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:11.887658 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ffnfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 13:01:11.887808 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:11.887714 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 13:01:12.838262 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:12.838227 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-r64l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 13:01:12.838643 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:12.838286 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 13:01:12.887935 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:12.887905 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-ffnfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 02 13:01:12.888064 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:12.887956 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 02 13:01:14.529251 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:14.529214 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:14.529895 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:14.529870 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:14.532119 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:14.532096 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:01:14.695853 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:14.695820 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:14.696152 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:14.696126 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:14.696903 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:14.696884 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:01:15.412490 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:15.412454 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:20.349836 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:20.349801 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:25.322824 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:25.322794 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:25.732555 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:25.732523 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:28.718576 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:28.718525 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:28.968720 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:28.968640 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:29.372191 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:01:29.372103 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:01:30.216884 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:30.216850 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:32.916137 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:32.916108 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:35.190411 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:35.190376 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:40.087518 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:40.087372 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:41.364865 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:41.364833 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:41.612679 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:41.612643 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:44.992789 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:44.992723 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:49.822200 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:49.822157 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:54.792188 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:54.792148 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:01:59.376321 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:01:59.376280 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:01:59.630791 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:01:59.630700 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:04.554942 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:04.554908 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:07.095485 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:07.095449 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:07.342286 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:07.342246 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:09.405419 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:09.405377 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:14.289397 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:14.289353 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:19.192046 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:19.192015 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:20.240534 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:20.240482 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:20.487303 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:20.487271 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:21.022295 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:21.022263 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:24.098027 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:24.097990 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:26.106349 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:26.106321 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:27.384680 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:27.384641 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:27.637795 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:27.637718 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:27.888253 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:27.888178 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:28.135512 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:28.135477 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:28.993697 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:28.993653 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:29.246512 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:29.246419 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:29.371808 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:02:29.371771 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:02:33.145874 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:33.145840 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:33.901829 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:33.901795 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:35.782778 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:35.782731 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:38.790629 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:38.790600 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:43.695483 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:43.695442 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:48.594423 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:48.594392 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:53.481898 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:53.481863 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:58.316270 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:02:58.316228 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:02:59.371930 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:02:59.371897 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:03:03.207532 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:03.207479 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:08.106098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:08.106018 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:13.021738 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:13.021697 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:17.887843 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:17.887804 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:22.779937 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:22.779895 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:26.428876 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:26.428838 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:27.725004 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:27.724968 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:29.373012 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:03:29.372973 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:03:32.587438 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:32.587402 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:37.419038 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:37.419007 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:42.304678 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:42.304643 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:43.069076 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:43.068990 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:43.319795 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:43.319714 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:47.277340 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:47.277297 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:52.112958 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:52.112929 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:56.990174 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:56.990139 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:58.727988 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:58.727945 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:58.972831 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:03:58.972801 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:03:59.372127 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:03:59.372097 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:04:01.891129 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:01.891100 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:06.787124 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:06.787089 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:11.684650 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:11.684610 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:16.585548 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:16.585515 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:21.495030 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:21.494994 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:26.396233 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:26.396195 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:26.770088 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:26.770059 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:29.372168 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:04:29.372132 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:04:31.290931 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:31.290895 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:36.191411 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:36.191311 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:38.709996 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:38.709954 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:41.087821 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:41.087789 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:45.997818 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:45.997787 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:50.884599 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:50.884570 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:55.777590 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:04:55.777548 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:04:59.372944 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:04:59.372907 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:05:00.694078 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:00.694041 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:05.579316 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:05.579276 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:10.497587 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:10.497554 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:15.390007 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:15.389971 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:20.292355 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:20.292321 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:25.184812 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:25.184774 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:27.115202 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:27.115174 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:29.373339 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:05:29.373302 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:05:30.091591 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:30.091561 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:34.982537 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:34.982484 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:39.881220 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:39.881191 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:44.794799 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:44.794765 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:49.684789 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:49.684758 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:54.578299 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:54.578268 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:56.836962 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:56.836929 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:57.083193 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:57.083158 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:05:59.372698 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:05:59.372662 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:05:59.480687 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:05:59.480652 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:04.306958 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:04.306922 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:09.190273 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:09.190193 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:14.106782 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:14.106752 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:18.991195 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:18.991163 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:21.923116 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:21.923065 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw"] Nov 02 13:06:21.923609 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:21.923362 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerName="istio-proxy" containerID="cri-o://de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404" gracePeriod=30 Nov 02 13:06:21.972556 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:21.972527 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5"] Nov 02 13:06:21.972809 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:21.972769 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerName="istio-proxy" containerID="cri-o://ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1" gracePeriod=30 Nov 02 13:06:23.898325 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:23.898286 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:27.219460 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.219435 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:06:27.242400 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.242381 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:06:27.264101 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264079 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istiod-ca-cert\") pod \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " Nov 02 13:06:27.264193 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264121 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-envoy\") pod \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " Nov 02 13:06:27.264193 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264172 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-credential-socket\") pod \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " Nov 02 13:06:27.264275 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264213 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-token\" (UniqueName: \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-token\") pod \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " Nov 02 13:06:27.264275 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264250 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-podinfo\") pod \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " Nov 02 13:06:27.264376 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264285 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-certs\") pod \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " Nov 02 13:06:27.264376 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264323 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-socket\") pod \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " Nov 02 13:06:27.264376 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264354 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-kube-api-access-hvtpn\" (UniqueName: \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-kube-api-access-hvtpn\") pod \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " Nov 02 13:06:27.264563 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264418 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-data\") pod \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\" (UID: \"63ad9289-bd34-4176-a6be-1f0b348ee3e1\") " Nov 02 13:06:27.264563 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264489 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-credential-socket" pod "63ad9289-bd34-4176-a6be-1f0b348ee3e1" (UID: "63ad9289-bd34-4176-a6be-1f0b348ee3e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:06:27.264563 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264537 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istiod-ca-cert" pod "63ad9289-bd34-4176-a6be-1f0b348ee3e1" (UID: "63ad9289-bd34-4176-a6be-1f0b348ee3e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:06:27.264740 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264634 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-socket" pod "63ad9289-bd34-4176-a6be-1f0b348ee3e1" (UID: "63ad9289-bd34-4176-a6be-1f0b348ee3e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:06:27.264798 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264754 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-certs" pod "63ad9289-bd34-4176-a6be-1f0b348ee3e1" (UID: "63ad9289-bd34-4176-a6be-1f0b348ee3e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:06:27.264798 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264765 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-data" pod "63ad9289-bd34-4176-a6be-1f0b348ee3e1" (UID: "63ad9289-bd34-4176-a6be-1f0b348ee3e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:06:27.264909 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264881 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.264909 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264898 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.265022 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264911 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.265022 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264924 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.265022 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.264938 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.266398 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.266370 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-envoy" pod "63ad9289-bd34-4176-a6be-1f0b348ee3e1" (UID: "63ad9289-bd34-4176-a6be-1f0b348ee3e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:06:27.266486 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.266415 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-podinfo" pod "63ad9289-bd34-4176-a6be-1f0b348ee3e1" (UID: "63ad9289-bd34-4176-a6be-1f0b348ee3e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 13:06:27.266486 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.266432 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-kube-api-access-hvtpn" pod "63ad9289-bd34-4176-a6be-1f0b348ee3e1" (UID: "63ad9289-bd34-4176-a6be-1f0b348ee3e1"). InnerVolumeSpecName "kube-api-access-hvtpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:06:27.266486 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.266420 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-token" pod "63ad9289-bd34-4176-a6be-1f0b348ee3e1" (UID: "63ad9289-bd34-4176-a6be-1f0b348ee3e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:06:27.366141 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366110 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-envoy\") pod \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " Nov 02 13:06:27.366290 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366150 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-kube-api-access-4skq8\" (UniqueName: \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-kube-api-access-4skq8\") pod \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " Nov 02 13:06:27.366290 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366220 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-credential-socket\") pod \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " Nov 02 13:06:27.366290 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366268 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-token\" (UniqueName: \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-token\") pod \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " Nov 02 13:06:27.366290 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366286 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-podinfo\") pod \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " Nov 02 13:06:27.366521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366312 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istiod-ca-cert\") pod \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " Nov 02 13:06:27.366521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366336 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-certs\") pod \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " Nov 02 13:06:27.366521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366375 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-socket\") pod \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " Nov 02 13:06:27.366521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366418 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-data\") pod \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\" (UID: \"85d962c8-a8b5-48e0-a6af-4162f5932d1d\") " Nov 02 13:06:27.366750 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366706 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.366750 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366726 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.366750 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366743 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63ad9289-bd34-4176-a6be-1f0b348ee3e1-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.366914 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366763 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvtpn\" (UniqueName: \"kubernetes.io/projected/63ad9289-bd34-4176-a6be-1f0b348ee3e1-kube-api-access-hvtpn\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.366914 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366785 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istiod-ca-cert" pod "85d962c8-a8b5-48e0-a6af-4162f5932d1d" (UID: "85d962c8-a8b5-48e0-a6af-4162f5932d1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:06:27.367022 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.366981 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-data" pod "85d962c8-a8b5-48e0-a6af-4162f5932d1d" (UID: "85d962c8-a8b5-48e0-a6af-4162f5932d1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:06:27.367240 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.367206 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-socket" pod "85d962c8-a8b5-48e0-a6af-4162f5932d1d" (UID: "85d962c8-a8b5-48e0-a6af-4162f5932d1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:06:27.367337 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.367308 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-certs" pod "85d962c8-a8b5-48e0-a6af-4162f5932d1d" (UID: "85d962c8-a8b5-48e0-a6af-4162f5932d1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:06:27.367533 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.367484 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-credential-socket" pod "85d962c8-a8b5-48e0-a6af-4162f5932d1d" (UID: "85d962c8-a8b5-48e0-a6af-4162f5932d1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:06:27.368651 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.368630 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-token" pod "85d962c8-a8b5-48e0-a6af-4162f5932d1d" (UID: "85d962c8-a8b5-48e0-a6af-4162f5932d1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:06:27.368651 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.368643 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-kube-api-access-4skq8" pod "85d962c8-a8b5-48e0-a6af-4162f5932d1d" (UID: "85d962c8-a8b5-48e0-a6af-4162f5932d1d"). InnerVolumeSpecName "kube-api-access-4skq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:06:27.368768 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.368744 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-podinfo" pod "85d962c8-a8b5-48e0-a6af-4162f5932d1d" (UID: "85d962c8-a8b5-48e0-a6af-4162f5932d1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 13:06:27.368870 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.368854 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-envoy" pod "85d962c8-a8b5-48e0-a6af-4162f5932d1d" (UID: "85d962c8-a8b5-48e0-a6af-4162f5932d1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:06:27.467382 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.467316 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.467382 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.467350 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.467382 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.467366 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.467583 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.467383 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.467583 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.467398 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.467583 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.467414 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.467583 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.467428 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.467583 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.467443 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d962c8-a8b5-48e0-a6af-4162f5932d1d-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.467583 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.467457 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4skq8\" (UniqueName: \"kubernetes.io/projected/85d962c8-a8b5-48e0-a6af-4162f5932d1d-kube-api-access-4skq8\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:06:27.469930 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.469902 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:27.953971 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.953943 4312 generic.go:356] "Generic (PLEG): container finished" podID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerID="de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404" exitCode=0 Nov 02 13:06:27.954108 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.953992 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" Nov 02 13:06:27.954108 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.954025 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" event={"ID":"63ad9289-bd34-4176-a6be-1f0b348ee3e1","Type":"ContainerDied","Data":"de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404"} Nov 02 13:06:27.954108 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.954081 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw" event={"ID":"63ad9289-bd34-4176-a6be-1f0b348ee3e1","Type":"ContainerDied","Data":"1654c20fcac0be69354d282628cde882f4316b61968695cc0285f1dadceb25a5"} Nov 02 13:06:27.954261 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.954115 4312 scope.go:122] "RemoveContainer" containerID="de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404" Nov 02 13:06:27.955467 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.955443 4312 generic.go:356] "Generic (PLEG): container finished" podID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerID="ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1" exitCode=0 Nov 02 13:06:27.955590 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.955469 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" event={"ID":"85d962c8-a8b5-48e0-a6af-4162f5932d1d","Type":"ContainerDied","Data":"ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1"} Nov 02 13:06:27.955590 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.955529 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" Nov 02 13:06:27.955689 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.955533 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5" event={"ID":"85d962c8-a8b5-48e0-a6af-4162f5932d1d","Type":"ContainerDied","Data":"715795f459a3e15c557927b1b08ecce4fd19742d890cbd51ecfb29cba2ce9a91"} Nov 02 13:06:27.965287 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.965270 4312 scope.go:122] "RemoveContainer" containerID="de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404" Nov 02 13:06:27.965590 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:06:27.965570 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404\": container with ID starting with de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404 not found: ID does not exist" containerID="de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404" Nov 02 13:06:27.965658 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.965598 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404"} err="failed to get container status \"de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404\": rpc error: code = NotFound desc = could not find container \"de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404\": container with ID starting with de55e79b12b93ecd4b19e2f3ecd9d8e09d083996f4636b88d6e40ed507a19404 not found: ID does not exist" Nov 02 13:06:27.965658 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.965615 4312 scope.go:122] "RemoveContainer" containerID="ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1" Nov 02 13:06:27.974724 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.974683 4312 scope.go:122] "RemoveContainer" containerID="ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1" Nov 02 13:06:27.975035 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:06:27.975009 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1\": container with ID starting with ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1 not found: ID does not exist" containerID="ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1" Nov 02 13:06:27.975111 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:27.975047 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1"} err="failed to get container status \"ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1\": rpc error: code = NotFound desc = could not find container \"ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1\": container with ID starting with ba24c8c5af1b6053ff73997e3e8a726c2473b4088345db423abffa05b70024c1 not found: ID does not exist" Nov 02 13:06:28.118136 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:28.118114 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw"] Nov 02 13:06:28.167450 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:28.167428 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-ffnfw"] Nov 02 13:06:28.234011 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:28.233959 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5"] Nov 02 13:06:28.240291 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:28.240270 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-r64l5"] Nov 02 13:06:28.789549 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:28.789518 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:29.074623 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:29.074554 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" path="/var/lib/kubelet/pods/63ad9289-bd34-4176-a6be-1f0b348ee3e1/volumes" Nov 02 13:06:29.075024 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:29.075010 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" path="/var/lib/kubelet/pods/85d962c8-a8b5-48e0-a6af-4162f5932d1d/volumes" Nov 02 13:06:29.372467 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:06:29.372381 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:06:33.602721 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:33.602685 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:38.489850 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:38.489820 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:43.397858 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:43.397824 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:48.281843 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:48.281807 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:53.086209 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:53.086171 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:58.096412 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:06:58.096375 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:06:59.377848 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:06:59.377804 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:07:02.996080 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:02.996024 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:07.944406 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:07.944371 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:12.815783 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:12.815747 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:17.696691 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:17.696658 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:22.593507 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:22.593468 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:27.486858 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:27.486820 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:27.810433 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:27.810343 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:29.371328 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:07:29.371293 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:07:30.364282 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:30.364247 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:30.691236 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:30.691191 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:30.951914 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:30.951808 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:32.383073 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:32.382988 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:37.423553 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:37.423396 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:42.374125 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:42.374089 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:42.622206 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:42.622175 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:42.869053 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:42.869015 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:47.206085 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:47.206043 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:53.133764 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:53.133712 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:53.378450 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:53.378397 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:56.987643 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:07:56.987611 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:07:59.372244 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:07:59.372206 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:08:01.891482 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:01.891425 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:06.782246 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:06.782180 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:11.692306 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:11.692268 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:12.219953 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:12.219917 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:12.470060 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:12.469951 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:16.516110 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:16.516076 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:21.384134 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:21.384100 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:26.304786 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:26.304752 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:28.159662 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:28.159621 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:29.374077 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:08:29.374038 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:08:31.210190 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:31.210152 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:36.094308 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:36.094276 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:41.087657 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:41.087623 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:45.992244 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:45.992210 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:50.896038 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:50.895992 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:55.790803 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:08:55.790764 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:08:59.373587 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:08:59.373542 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:09:00.687130 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:00.687098 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:05.509019 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:05.508952 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:10.393158 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:10.393095 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:15.286590 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:15.286554 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:20.107807 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:20.107774 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:24.983637 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:24.983557 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:28.501276 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:28.501235 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:29.372041 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:09:29.371999 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:09:29.894722 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:29.894683 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:34.793459 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:34.793425 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:38.458557 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:38.458521 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955"] Nov 02 13:09:38.459143 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:38.458757 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" containerID="cri-o://2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c" gracePeriod=30 Nov 02 13:09:39.688383 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:39.688312 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:43.698346 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.698322 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:09:43.788715 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.788657 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-token\" (UniqueName: \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-token\") pod \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " Nov 02 13:09:43.788715 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.788695 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-certs\") pod \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " Nov 02 13:09:43.788715 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.788715 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-envoy\") pod \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " Nov 02 13:09:43.788962 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.788742 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-socket\") pod \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " Nov 02 13:09:43.788962 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.788780 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-credential-socket\") pod \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " Nov 02 13:09:43.788962 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.788835 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istiod-ca-cert\") pod \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " Nov 02 13:09:43.788962 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.788861 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-podinfo\") pod \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " Nov 02 13:09:43.788962 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.788897 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-kube-api-access-fs9z7\" (UniqueName: \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-kube-api-access-fs9z7\") pod \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " Nov 02 13:09:43.788962 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.788923 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-certs" pod "8ea83810-a6ec-4e8a-b4b2-00457e869a8f" (UID: "8ea83810-a6ec-4e8a-b4b2-00457e869a8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:09:43.788962 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.788944 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-data\") pod \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\" (UID: \"8ea83810-a6ec-4e8a-b4b2-00457e869a8f\") " Nov 02 13:09:43.789345 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.789184 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:09:43.789345 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.789204 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-credential-socket" pod "8ea83810-a6ec-4e8a-b4b2-00457e869a8f" (UID: "8ea83810-a6ec-4e8a-b4b2-00457e869a8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:09:43.789345 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.789293 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istiod-ca-cert" pod "8ea83810-a6ec-4e8a-b4b2-00457e869a8f" (UID: "8ea83810-a6ec-4e8a-b4b2-00457e869a8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:09:43.789542 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.789408 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-data" pod "8ea83810-a6ec-4e8a-b4b2-00457e869a8f" (UID: "8ea83810-a6ec-4e8a-b4b2-00457e869a8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:09:43.789542 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.789425 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-socket" pod "8ea83810-a6ec-4e8a-b4b2-00457e869a8f" (UID: "8ea83810-a6ec-4e8a-b4b2-00457e869a8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:09:43.791006 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.790982 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-token" pod "8ea83810-a6ec-4e8a-b4b2-00457e869a8f" (UID: "8ea83810-a6ec-4e8a-b4b2-00457e869a8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:09:43.791121 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.791100 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-podinfo" pod "8ea83810-a6ec-4e8a-b4b2-00457e869a8f" (UID: "8ea83810-a6ec-4e8a-b4b2-00457e869a8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 13:09:43.791316 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.791289 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-envoy" pod "8ea83810-a6ec-4e8a-b4b2-00457e869a8f" (UID: "8ea83810-a6ec-4e8a-b4b2-00457e869a8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:09:43.791376 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.791357 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-kube-api-access-fs9z7" pod "8ea83810-a6ec-4e8a-b4b2-00457e869a8f" (UID: "8ea83810-a6ec-4e8a-b4b2-00457e869a8f"). InnerVolumeSpecName "kube-api-access-fs9z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:09:43.814066 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.814042 4312 generic.go:356] "Generic (PLEG): container finished" podID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerID="2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c" exitCode=0 Nov 02 13:09:43.814177 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.814086 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" Nov 02 13:09:43.814177 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.814130 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" event={"ID":"8ea83810-a6ec-4e8a-b4b2-00457e869a8f","Type":"ContainerDied","Data":"2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c"} Nov 02 13:09:43.814297 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.814194 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955" event={"ID":"8ea83810-a6ec-4e8a-b4b2-00457e869a8f","Type":"ContainerDied","Data":"2cf51ce3edbe4927177f5f8ae95e1c236bbf45f2dcb8cef2aaa51ccdbaa1cdf5"} Nov 02 13:09:43.814297 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.814227 4312 scope.go:122] "RemoveContainer" containerID="2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c" Nov 02 13:09:43.824488 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.824475 4312 scope.go:122] "RemoveContainer" containerID="2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c" Nov 02 13:09:43.824943 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:09:43.824877 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c\": container with ID starting with 2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c not found: ID does not exist" containerID="2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c" Nov 02 13:09:43.825326 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.824910 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c"} err="failed to get container status \"2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c\": rpc error: code = NotFound desc = could not find container \"2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c\": container with ID starting with 2b26ef2e13c4689889e1e31ab13d1d4e93fbc95af098f9198c42b6000a244c0c not found: ID does not exist" Nov 02 13:09:43.889721 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.889691 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:09:43.889721 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.889720 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:09:43.889721 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.889731 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:09:43.889892 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.889740 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs9z7\" (UniqueName: \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-kube-api-access-fs9z7\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:09:43.889892 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.889751 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:09:43.889892 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.889759 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:09:43.889892 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.889767 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:09:43.889892 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.889775 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ea83810-a6ec-4e8a-b4b2-00457e869a8f-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:09:43.983313 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.983283 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955"] Nov 02 13:09:43.989685 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:43.989662 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-kj955"] Nov 02 13:09:44.506161 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:44.506129 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:45.074474 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:45.074437 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" path="/var/lib/kubelet/pods/8ea83810-a6ec-4e8a-b4b2-00457e869a8f/volumes" Nov 02 13:09:49.402675 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:49.402645 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:54.291972 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:54.291934 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:59.205670 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:09:59.205628 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:09:59.372340 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:09:59.372301 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:10:02.612294 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:02.612250 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:02.863800 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:02.863715 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:04.109323 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:04.109293 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:09.012663 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:09.012622 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:13.869991 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:13.869952 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:18.777716 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:18.777664 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:19.967457 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:19.967394 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:20.212531 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:20.212456 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:23.689873 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:23.689830 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:23.948995 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:23.948905 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:28.574589 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:28.574540 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:28.863255 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:28.863145 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:29.373635 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:10:29.373597 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:10:33.403574 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:33.403542 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:38.291057 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:38.290956 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:43.180882 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:43.180839 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:47.999080 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:47.999050 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:52.890687 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:52.890647 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:57.775831 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:10:57.775784 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:10:59.374960 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:10:59.374905 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:11:02.598112 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:02.598076 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:07.509278 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:07.509246 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:12.399516 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:12.399470 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:17.214356 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:17.214319 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:22.078110 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:22.078075 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:26.887879 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:26.887846 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:29.188514 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:29.188459 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:29.372392 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:11:29.372360 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:11:31.780917 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:31.780887 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:36.682327 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:36.682282 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:41.575796 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:41.575758 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:46.390073 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:46.390037 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:51.295058 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:51.295022 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:56.095472 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:11:56.095435 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:11:59.373911 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:11:59.373872 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:12:01.018684 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:01.018652 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:05.798920 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:05.798835 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:06.046066 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:06.046024 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:06.654355 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:06.654319 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:09.167676 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:09.167633 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:09.412485 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:09.412453 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:10.792872 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:10.792833 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:15.600958 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:15.600922 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:20.587075 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:20.587031 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:22.282996 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:22.282961 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:25.498043 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:25.498010 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:29.375367 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:12:29.375326 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:12:29.536463 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:29.536427 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:30.324878 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:30.324847 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:34.373001 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:34.372961 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:34.634664 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:34.634587 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:35.209915 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:35.209882 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:40.093245 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:40.093206 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:44.905406 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:44.905373 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:49.788993 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:49.788957 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:54.685237 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:54.685203 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:59.372123 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:12:59.372089 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:12:59.588171 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:59.588143 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:12:59.839300 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:12:59.839267 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:00.084455 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:00.084428 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:00.573074 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:00.573035 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:02.596992 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:02.596958 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:04.493849 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:04.493813 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:06.906196 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:06.906164 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:07.156777 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:07.156702 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:07.405623 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:07.405587 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:09.389681 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:09.389650 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:13.961219 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:13.961181 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:14.318344 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:14.318257 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:15.148553 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:15.148519 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:15.396391 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:15.396359 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:18.786747 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:18.786712 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:19.921210 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:19.921178 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:20.597080 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:20.597045 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:22.585162 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:22.585119 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:25.477925 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:25.477881 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:25.731038 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:25.730956 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:25.982549 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:25.982465 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:27.322672 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:27.322638 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:27.981249 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:27.981213 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:28.712779 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:28.712734 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:29.372515 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:13:29.372473 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:13:29.876035 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:29.875999 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:30.850491 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:30.850457 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:35.222834 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:35.222791 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:36.814481 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:36.814410 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:37.058677 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:37.058638 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:40.247537 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.247488 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb"] Nov 02 13:13:40.247959 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.247695 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" podUID="6ac0f0a6-22c8-465b-99c9-d6028080ce42" containerName="maas-api" containerID="cri-o://02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232" gracePeriod=30 Nov 02 13:13:40.450997 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.450966 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:40.483215 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.483194 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 13:13:40.626744 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.626680 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-maas-api-tls\") pod \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " Nov 02 13:13:40.626744 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.626712 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-metrics-certs\") pod \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " Nov 02 13:13:40.626744 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.626739 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac0f0a6-22c8-465b-99c9-d6028080ce42-kube-api-access-zdszq\" (UniqueName: \"kubernetes.io/projected/6ac0f0a6-22c8-465b-99c9-d6028080ce42-kube-api-access-zdszq\") pod \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " Nov 02 13:13:40.626991 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.626804 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac0f0a6-22c8-465b-99c9-d6028080ce42-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0f0a6-22c8-465b-99c9-d6028080ce42-openshift-service-ca\") pod \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\" (UID: \"6ac0f0a6-22c8-465b-99c9-d6028080ce42\") " Nov 02 13:13:40.627241 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.627214 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac0f0a6-22c8-465b-99c9-d6028080ce42-openshift-service-ca" pod "6ac0f0a6-22c8-465b-99c9-d6028080ce42" (UID: "6ac0f0a6-22c8-465b-99c9-d6028080ce42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:13:40.628801 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.628774 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-metrics-certs" pod "6ac0f0a6-22c8-465b-99c9-d6028080ce42" (UID: "6ac0f0a6-22c8-465b-99c9-d6028080ce42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:13:40.628918 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.628799 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-maas-api-tls" pod "6ac0f0a6-22c8-465b-99c9-d6028080ce42" (UID: "6ac0f0a6-22c8-465b-99c9-d6028080ce42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:13:40.628918 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.628812 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac0f0a6-22c8-465b-99c9-d6028080ce42-kube-api-access-zdszq" pod "6ac0f0a6-22c8-465b-99c9-d6028080ce42" (UID: "6ac0f0a6-22c8-465b-99c9-d6028080ce42"). InnerVolumeSpecName "kube-api-access-zdszq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:13:40.727558 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.727536 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-maas-api-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:13:40.727558 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.727557 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac0f0a6-22c8-465b-99c9-d6028080ce42-metrics-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:13:40.727690 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.727567 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdszq\" (UniqueName: \"kubernetes.io/projected/6ac0f0a6-22c8-465b-99c9-d6028080ce42-kube-api-access-zdszq\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:13:40.727690 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.727577 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0f0a6-22c8-465b-99c9-d6028080ce42-openshift-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:13:40.855293 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.855265 4312 generic.go:356] "Generic (PLEG): container finished" podID="6ac0f0a6-22c8-465b-99c9-d6028080ce42" containerID="02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232" exitCode=0 Nov 02 13:13:40.855402 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.855315 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" Nov 02 13:13:40.855402 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.855353 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" event={"ID":"6ac0f0a6-22c8-465b-99c9-d6028080ce42","Type":"ContainerDied","Data":"02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232"} Nov 02 13:13:40.855478 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.855409 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb" event={"ID":"6ac0f0a6-22c8-465b-99c9-d6028080ce42","Type":"ContainerDied","Data":"89fa85be8f6e380b68a0d7ad01e15ed79c3fe47c320c6cf655da461cce7ad3bd"} Nov 02 13:13:40.855478 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.855434 4312 scope.go:122] "RemoveContainer" containerID="02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232" Nov 02 13:13:40.864593 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.864571 4312 scope.go:122] "RemoveContainer" containerID="02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232" Nov 02 13:13:40.864834 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:13:40.864817 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232\": container with ID starting with 02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232 not found: ID does not exist" containerID="02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232" Nov 02 13:13:40.864881 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:40.864841 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232"} err="failed to get container status \"02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232\": rpc error: code = NotFound desc = could not find container \"02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232\": container with ID starting with 02cb5a2cb8028855a8de2d61f53e763ed8ba5e6a391854c8722a6ae6c8c9c232 not found: ID does not exist" Nov 02 13:13:41.018113 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:41.018087 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb"] Nov 02 13:13:41.024292 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:41.024272 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-3a78b4-7546fcd8bb-sqmsb"] Nov 02 13:13:41.074677 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:41.074647 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac0f0a6-22c8-465b-99c9-d6028080ce42" path="/var/lib/kubelet/pods/6ac0f0a6-22c8-465b-99c9-d6028080ce42/volumes" Nov 02 13:13:41.391186 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:41.391121 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:47.245309 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:47.245275 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:47.494809 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:47.494772 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:48.878564 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:48.878520 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:49.355513 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:49.355462 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:13:50.743659 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:50.743630 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wqxtt_39bf2d16-2a6d-4822-8d9c-b768de423729/manager/0.log" Nov 02 13:13:51.283548 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:51.283522 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-ck9gr_89fbd2ad-bb31-409d-90ba-b7d358be49a7/kuadrant-console-plugin/0.log" Nov 02 13:13:54.700201 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:54.700169 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vfxbz_fdf81983-c2ed-4444-9837-16ffca122340/manager/0.log" Nov 02 13:13:55.634599 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:13:55.634566 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7w4zh_5ced755a-4baf-4b4b-96df-f8a57839c593/server/0.log" Nov 02 13:13:59.371905 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:13:59.371872 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:14:00.856610 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:00.856576 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:01.490076 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:01.490040 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:01.742080 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:01.741997 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:04.959395 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:04.959359 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:05.214739 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:05.214669 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:06.742170 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:06.742143 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:06.990697 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:06.990668 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:08.255678 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:08.255642 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:08.528953 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:08.528875 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:11.529666 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:11.529633 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:11.784057 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:11.783985 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:16.508920 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:16.508875 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:21.515917 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:21.515881 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:26.406272 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:26.406234 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:29.372977 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:14:29.372941 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:14:30.224402 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:30.224366 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:31.779320 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:31.779289 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:36.898483 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:36.898443 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:41.792725 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:41.792664 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:46.623566 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:46.623525 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:51.483829 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:51.483791 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:53.356632 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:53.356599 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:53.602173 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:53.602139 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:56.282050 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:14:56.282009 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:14:59.373550 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:14:59.373516 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:15:00.564805 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:00.564774 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:15:02.160335 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:02.160296 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:15:02.442040 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:02.442007 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:15:07.501329 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:07.501252 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:15:29.373020 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:15:29.372982 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:15:30.570156 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:30.570124 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:15:38.537809 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:38.537773 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:15:38.787219 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:38.787188 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:15:41.248579 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248544 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2"] Nov 02 13:15:41.248955 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248851 4312 cpu_manager.go:414] "containerMap: removing container" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" Nov 02 13:15:41.248955 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248862 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" Nov 02 13:15:41.248955 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248875 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6ac0f0a6-22c8-465b-99c9-d6028080ce42" containerName="maas-api" Nov 02 13:15:41.248955 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248881 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac0f0a6-22c8-465b-99c9-d6028080ce42" containerName="maas-api" Nov 02 13:15:41.248955 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248891 4312 cpu_manager.go:414] "containerMap: removing container" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerName="istio-proxy" Nov 02 13:15:41.248955 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248897 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerName="istio-proxy" Nov 02 13:15:41.248955 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248912 4312 cpu_manager.go:414] "containerMap: removing container" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerName="istio-proxy" Nov 02 13:15:41.248955 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248917 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerName="istio-proxy" Nov 02 13:15:41.249247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248967 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d962c8-a8b5-48e0-a6af-4162f5932d1d" containerName="istio-proxy" Nov 02 13:15:41.249247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248976 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="63ad9289-bd34-4176-a6be-1f0b348ee3e1" containerName="istio-proxy" Nov 02 13:15:41.249247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248984 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea83810-a6ec-4e8a-b4b2-00457e869a8f" containerName="istio-proxy" Nov 02 13:15:41.249247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.248991 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac0f0a6-22c8-465b-99c9-d6028080ce42" containerName="maas-api" Nov 02 13:15:41.251981 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.251958 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.319300 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.319273 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.319445 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.319318 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.319445 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.319339 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.319445 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.319396 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8426f988-1fd5-442f-957d-1e1a58590ee0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.319655 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.319544 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.319655 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.319572 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.319655 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.319604 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.319752 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.319658 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g8ht\" (UniqueName: \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-kube-api-access-7g8ht\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.319752 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.319698 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.353621 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.353600 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c5ldd\"" Nov 02 13:15:41.353734 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.353640 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 13:15:41.404741 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.404720 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2"] Nov 02 13:15:41.420758 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.420735 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.420886 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.420766 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.420886 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.420785 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.420886 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.420804 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g8ht\" (UniqueName: \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-kube-api-access-7g8ht\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.420886 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.420835 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.421090 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.420879 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.421090 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.420942 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.421090 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.420976 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.421090 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.420999 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8426f988-1fd5-442f-957d-1e1a58590ee0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.421301 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.421207 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.421364 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.421302 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.421405 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.421360 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.421471 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.421449 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.421773 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.421755 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8426f988-1fd5-442f-957d-1e1a58590ee0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.423105 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.423089 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.423396 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.423375 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.504541 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.504481 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.519595 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.519566 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g8ht\" (UniqueName: \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-kube-api-access-7g8ht\") pod \"maas-default-gateway-openshift-default-6dcf757899-d9wv2\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.563565 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.563536 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:41.986310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.986282 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2"] Nov 02 13:15:41.988813 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:15:41.988788 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8426f988_1fd5_442f_957d_1e1a58590ee0.slice/crio-fc0c8de3c47a1ed645edc97283f3c03d29cba0e106e0d56b8fb1b445f8154262 WatchSource:0}: Error finding container fc0c8de3c47a1ed645edc97283f3c03d29cba0e106e0d56b8fb1b445f8154262: Status 404 returned error can't find the container with id fc0c8de3c47a1ed645edc97283f3c03d29cba0e106e0d56b8fb1b445f8154262 Nov 02 13:15:41.990883 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.990849 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:15:41.990963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.990916 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:15:41.990963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:41.990947 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:15:42.377588 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:42.377423 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" event={"ID":"8426f988-1fd5-442f-957d-1e1a58590ee0","Type":"ContainerStarted","Data":"d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259"} Nov 02 13:15:42.377588 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:42.377506 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" event={"ID":"8426f988-1fd5-442f-957d-1e1a58590ee0","Type":"ContainerStarted","Data":"fc0c8de3c47a1ed645edc97283f3c03d29cba0e106e0d56b8fb1b445f8154262"} Nov 02 13:15:42.568522 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:42.567646 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:42.595768 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:42.595717 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podStartSLOduration=1.595701315 podStartE2EDuration="1.595701315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:42.545694396 +0000 UTC m=+4759.575930646" watchObservedRunningTime="2026-11-02 13:15:42.595701315 +0000 UTC m=+4759.625937562" Nov 02 13:15:43.572533 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:43.569083 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d9wv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 13:15:43.572533 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:43.569136 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 13:15:44.572884 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:44.572846 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d9wv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 13:15:44.573296 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:44.572910 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 13:15:45.570524 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:45.570129 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d9wv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 13:15:45.570524 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:45.570185 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 13:15:45.688067 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:45.688032 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:15:46.572521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:46.571043 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d9wv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 13:15:46.572521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:46.571096 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 13:15:47.568521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:47.566906 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d9wv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 13:15:47.568521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:47.566979 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 13:15:47.930548 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:47.930517 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:15:48.564244 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:48.564209 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d9wv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 13:15:48.564433 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:48.564276 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 13:15:49.563907 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:49.563868 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d9wv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 13:15:49.564310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:49.563923 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 13:15:51.101635 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:51.101604 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:51.102156 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:51.102101 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:51.102691 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:15:51.102673 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:15:59.375562 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:15:59.375491 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:16:26.614171 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.614135 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6"] Nov 02 13:16:26.617449 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.617426 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.708764 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.708739 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.708876 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.708779 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.708876 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.708805 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.708962 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.708872 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.709024 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.709002 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.709068 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.709049 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.709116 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.709102 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpjpr\" (UniqueName: \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-kube-api-access-gpjpr\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.709162 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.709131 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.709162 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.709146 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.809800 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.809771 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.809959 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.809815 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpjpr\" (UniqueName: \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-kube-api-access-gpjpr\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.809959 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.809840 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.809959 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.809863 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.810149 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.809984 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.810149 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.810043 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.810149 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.810086 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.810149 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.810107 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.810149 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.810145 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.810414 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.810320 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.810674 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.810627 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.810920 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.810886 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.811036 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.810923 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.811036 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.810949 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.812299 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.812276 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.812653 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.812632 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.872043 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.871973 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6"] Nov 02 13:16:26.893765 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.893739 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.893940 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.893920 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpjpr\" (UniqueName: \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-kube-api-access-gpjpr\") pod \"maas-default-gateway-openshift-default-6dcf757899-tbpd6\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:26.929812 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:26.929790 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:27.350340 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:27.350313 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6"] Nov 02 13:16:27.353124 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:16:27.353088 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0fa5df_699a_4f85_a72d_bf6fb8836fb8.slice/crio-f00e38c41c9378a112b51ad0de5e6fac043d03ce878449c2743ea472ef947c59 WatchSource:0}: Error finding container f00e38c41c9378a112b51ad0de5e6fac043d03ce878449c2743ea472ef947c59: Status 404 returned error can't find the container with id f00e38c41c9378a112b51ad0de5e6fac043d03ce878449c2743ea472ef947c59 Nov 02 13:16:27.355584 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:27.355552 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:16:27.355668 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:27.355635 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:16:27.355708 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:27.355665 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:16:27.575748 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:27.575710 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" event={"ID":"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8","Type":"ContainerStarted","Data":"3147ebcab222d66dd839f4f503636c92c210d2fc084cee8ea569066f3b7c350b"} Nov 02 13:16:27.575915 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:27.575767 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" event={"ID":"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8","Type":"ContainerStarted","Data":"f00e38c41c9378a112b51ad0de5e6fac043d03ce878449c2743ea472ef947c59"} Nov 02 13:16:27.745473 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:27.745406 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podStartSLOduration=1.74538678 podStartE2EDuration="1.74538678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:16:27.695389674 +0000 UTC m=+4804.725625924" watchObservedRunningTime="2026-11-02 13:16:27.74538678 +0000 UTC m=+4804.775623032" Nov 02 13:16:27.930892 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:27.930854 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:28.932061 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:28.932027 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-tbpd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 13:16:28.932462 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:28.932082 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 13:16:29.393559 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:16:29.393465 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:16:29.930633 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:29.930603 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-tbpd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 13:16:29.930817 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:29.930653 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 13:16:30.931018 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:30.930986 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-tbpd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 13:16:30.931401 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:30.931038 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 13:16:30.942071 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:30.942043 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:16:31.930165 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:31.930135 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-tbpd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 13:16:31.930345 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:31.930191 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 13:16:32.931143 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:32.931109 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-tbpd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 13:16:32.931668 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:32.931164 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 13:16:33.930757 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:33.930722 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-tbpd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 13:16:33.930932 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:33.930771 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 13:16:34.930543 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:34.930488 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-tbpd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 13:16:34.930933 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:34.930560 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 13:16:36.932179 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:36.932085 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-tbpd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:16:36.932616 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:36.932169 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:16:37.973659 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:37.973621 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-tbpd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:16:37.974220 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:37.974190 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:16:39.016653 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:39.016610 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-tbpd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:16:39.017046 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:39.016674 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:16:39.451998 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:39.451971 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:39.452269 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:39.452245 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:39.452882 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:39.452863 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:16:44.933251 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:44.933210 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:16:45.183055 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:45.183024 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:16:49.133653 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:49.133620 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:16:56.836876 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.836845 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95"] Nov 02 13:16:56.840375 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.840359 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.892883 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.892855 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf62p\" (UniqueName: \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-kube-api-access-sf62p\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.892997 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.892900 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.893041 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.893004 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.893078 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.893059 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.893151 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.893090 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.893151 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.893108 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc38d7e-c396-43ac-83dc-76fa63725a08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.893151 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.893129 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.893151 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.893147 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.893299 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.893285 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.946376 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.946347 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95"] Nov 02 13:16:56.994024 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.993999 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994145 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994070 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf62p\" (UniqueName: \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-kube-api-access-sf62p\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994145 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994104 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994145 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994130 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994316 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994161 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994316 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994205 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994316 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994226 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc38d7e-c396-43ac-83dc-76fa63725a08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994316 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994254 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994316 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994276 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994615 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994316 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994615 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994447 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994763 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994644 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994763 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994753 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.994973 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.994948 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc38d7e-c396-43ac-83dc-76fa63725a08-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.996391 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.996369 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:56.996832 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:56.996813 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:57.054895 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:57.054873 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:57.055078 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:57.055061 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf62p\" (UniqueName: \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-kube-api-access-sf62p\") pod \"maas-default-gateway-openshift-default-6dcf757899-8gm95\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:57.152818 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:57.152766 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:57.576245 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:57.576160 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95"] Nov 02 13:16:57.578468 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:16:57.578442 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc38d7e_c396_43ac_83dc_76fa63725a08.slice/crio-ea171cb93e557d3bf89e1fe65252a1d6f780ba3c4877113ec28219d47ed6b362 WatchSource:0}: Error finding container ea171cb93e557d3bf89e1fe65252a1d6f780ba3c4877113ec28219d47ed6b362: Status 404 returned error can't find the container with id ea171cb93e557d3bf89e1fe65252a1d6f780ba3c4877113ec28219d47ed6b362 Nov 02 13:16:57.580593 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:57.580559 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:16:57.580724 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:57.580637 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:16:57.580724 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:57.580683 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:16:57.733462 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:57.733419 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" event={"ID":"3fc38d7e-c396-43ac-83dc-76fa63725a08","Type":"ContainerStarted","Data":"3ce2099be6094c1300b462afbe8b9cdb464acfe027a9120ccf55125375514829"} Nov 02 13:16:57.733670 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:57.733491 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" event={"ID":"3fc38d7e-c396-43ac-83dc-76fa63725a08","Type":"ContainerStarted","Data":"ea171cb93e557d3bf89e1fe65252a1d6f780ba3c4877113ec28219d47ed6b362"} Nov 02 13:16:57.904022 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:57.903917 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podStartSLOduration=1.903903515 podStartE2EDuration="1.903903515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:16:57.852965621 +0000 UTC m=+4834.883201872" watchObservedRunningTime="2026-11-02 13:16:57.903903515 +0000 UTC m=+4834.934139765" Nov 02 13:16:58.166586 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:58.159875 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:16:58.172694 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:58.172652 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8gm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 13:16:58.172886 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:58.172721 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 13:16:59.168521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:59.167836 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8gm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 13:16:59.168521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:16:59.167892 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 13:16:59.392736 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:16:59.392702 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:17:00.168524 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:00.162856 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8gm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 13:17:00.168524 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:00.162913 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 13:17:01.172521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:01.170180 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8gm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 13:17:01.172521 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:01.170235 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 13:17:02.165525 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:02.159832 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8gm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 13:17:02.165525 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:02.159888 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 13:17:03.158531 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:03.157878 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8gm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 13:17:03.158531 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:03.157939 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 13:17:04.153880 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:04.153846 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8gm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 13:17:04.154130 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:04.153898 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 13:17:05.153333 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:05.153299 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8gm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 02 13:17:05.153836 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:05.153354 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 02 13:17:07.154227 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:07.154191 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8gm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:17:07.154688 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:07.154238 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:17:08.195674 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:08.195627 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8gm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:17:08.196195 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:08.195707 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:17:09.236667 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:09.236629 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-8gm95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:17:09.237050 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:09.236693 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:17:09.659915 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:09.659819 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:17:09.660168 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:09.660139 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:17:09.660696 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:09.660677 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:17:29.373215 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:17:29.373177 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:17:31.347300 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:31.347261 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:17:32.767650 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:32.767597 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:17:41.878202 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:41.878164 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:17:45.286694 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:45.286642 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:17:46.026932 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:17:46.026900 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:17:59.373074 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:17:59.373037 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:18:08.860445 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:18:08.860357 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:18:29.372777 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:18:29.372736 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:18:31.775522 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:18:31.772305 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:18:55.435226 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:18:55.435192 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:18:55.699907 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:18:55.699827 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:18:59.373145 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:18:59.373110 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:19:13.611425 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:13.611391 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:14.326332 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:14.326300 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:14.704802 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:14.704767 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:18.316820 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:18.316769 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:18.609336 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:18.609246 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:21.805117 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:21.805062 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:22.124929 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:22.124849 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:25.294028 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:25.293992 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:25.579632 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:25.579542 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:25.855056 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:25.854973 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:29.371711 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:19:29.371667 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:19:30.648204 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:30.648166 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:32.026102 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:32.026066 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:35.621986 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:35.621900 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:40.721772 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:40.721738 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:44.632920 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:44.632888 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:45.002735 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:45.002700 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:46.310662 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:46.310618 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:48.110426 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:48.110391 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:51.200013 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:51.199974 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:56.239741 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:19:56.239710 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:19:59.373421 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:19:59.373383 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:20:01.313068 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:01.313038 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:03.649558 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:03.649521 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:06.312666 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:06.312634 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:11.512285 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:11.512245 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:16.426218 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:16.426176 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:21.554161 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:21.554126 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:26.504627 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:26.504593 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:29.375989 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:20:29.375936 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:20:31.418000 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:31.417967 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:32.406357 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:32.406320 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:36.420671 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:36.420632 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:41.300227 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:41.300195 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:46.335125 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:46.335085 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:48.692168 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:48.692129 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:49.181004 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:49.180967 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:51.300605 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:51.300572 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:53.741790 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:53.741751 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:55.374132 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:55.374094 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:55.812486 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:55.812453 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:56.073260 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:56.073178 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:56.334584 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:20:56.334484 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:20:59.374348 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:20:59.374307 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:21:01.535218 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:01.535181 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:01.797835 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:01.797762 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:29.374867 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:21:29.374786 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:21:30.487482 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:30.487448 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:30.745146 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:30.745064 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:30.991700 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:30.991648 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:31.254286 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:31.254253 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:32.719381 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:32.719344 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:33.073987 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:33.073909 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:37.163230 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:37.163185 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:37.408780 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:37.408747 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:37.996263 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:37.996221 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:42.907753 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:42.907714 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:47.684691 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:47.684649 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:52.603394 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:52.603360 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:57.404891 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:57.404859 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:57.777186 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:21:57.777148 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:21:59.376017 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:21:59.375976 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:22:01.689534 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:22:01.689473 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:22:03.231361 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:22:03.231324 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:22:03.517782 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:22:03.517699 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:22:04.185001 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:22:04.184958 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:22:10.768760 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:22:10.768726 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:22:11.024174 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:22:11.024094 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:22:11.275604 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:22:11.275525 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:22:29.373437 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:22:29.373395 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:22:33.098659 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:22:33.098622 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:22:37.923547 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:22:37.923441 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:22:59.371826 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:22:59.371789 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:23:20.549425 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:23:20.549387 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:23:28.342816 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:23:28.342778 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:23:29.383719 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:23:29.383679 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:23:33.382744 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:23:33.382709 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:23:37.342219 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:23:37.342189 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:23:39.204754 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:23:39.204721 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:23:59.371686 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:23:59.371649 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:24:24.712204 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:24.712120 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:25.314061 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:25.314026 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:25.584035 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:25.583951 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:29.372055 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:24:29.372021 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:24:29.576410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:29.576381 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:29.824084 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:29.824048 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:32.884052 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:32.884006 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:33.137898 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:33.137807 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:33.725975 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:33.725936 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:34.420183 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:34.420148 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:34.667960 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:34.667921 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:41.616638 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:41.616603 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:41.862794 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:41.862758 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:42.118132 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:42.118098 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:45.428778 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:45.428725 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:45.926630 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:45.926601 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:48.513184 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:48.513149 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:53.725720 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:24:53.725685 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:24:59.374252 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:24:59.374215 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:25:00.428692 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:00.428656 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:05.076652 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:05.076618 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:05.975892 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:05.975854 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:06.234597 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:06.234523 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:06.639109 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:06.639032 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:06.887657 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:06.887624 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:10.570807 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:10.570768 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:10.812607 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:10.812573 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:13.050929 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:13.050891 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:13.870808 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:13.870774 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:14.123696 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.123616 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:14.518801 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.518771 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6"] Nov 02 13:25:14.522042 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.522025 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" Nov 02 13:25:14.623760 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.623731 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-193883-dockercfg-xdh52\"" Nov 02 13:25:14.623760 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.623754 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-193883\"" Nov 02 13:25:14.625351 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.625333 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6"] Nov 02 13:25:14.629431 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.629414 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb68ab36-8c03-4ddc-b363-12f65361b836-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6\" (UID: \"cb68ab36-8c03-4ddc-b363-12f65361b836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" Nov 02 13:25:14.629484 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.629443 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fws8\" (UniqueName: \"kubernetes.io/projected/cb68ab36-8c03-4ddc-b363-12f65361b836-kube-api-access-5fws8\") pod \"payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6\" (UID: \"cb68ab36-8c03-4ddc-b363-12f65361b836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" Nov 02 13:25:14.729887 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.729860 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb68ab36-8c03-4ddc-b363-12f65361b836-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6\" (UID: \"cb68ab36-8c03-4ddc-b363-12f65361b836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" Nov 02 13:25:14.730020 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.729893 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fws8\" (UniqueName: \"kubernetes.io/projected/cb68ab36-8c03-4ddc-b363-12f65361b836-kube-api-access-5fws8\") pod \"payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6\" (UID: \"cb68ab36-8c03-4ddc-b363-12f65361b836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" Nov 02 13:25:14.730618 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.730599 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb68ab36-8c03-4ddc-b363-12f65361b836-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6\" (UID: \"cb68ab36-8c03-4ddc-b363-12f65361b836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" Nov 02 13:25:14.790849 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.790792 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fws8\" (UniqueName: \"kubernetes.io/projected/cb68ab36-8c03-4ddc-b363-12f65361b836-kube-api-access-5fws8\") pod \"payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6\" (UID: \"cb68ab36-8c03-4ddc-b363-12f65361b836\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" Nov 02 13:25:14.834742 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:14.834722 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" Nov 02 13:25:15.081821 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:15.081753 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6"] Nov 02 13:25:15.205443 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:15.205374 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6"] Nov 02 13:25:15.208379 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:25:15.208353 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb68ab36_8c03_4ddc_b363_12f65361b836.slice/crio-7ed738cee9cf71b2d98ec2b03bf53e1d8cf5a8685bdae541d2fc3619ed0ea344 WatchSource:0}: Error finding container 7ed738cee9cf71b2d98ec2b03bf53e1d8cf5a8685bdae541d2fc3619ed0ea344: Status 404 returned error can't find the container with id 7ed738cee9cf71b2d98ec2b03bf53e1d8cf5a8685bdae541d2fc3619ed0ea344 Nov 02 13:25:15.244852 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:15.244825 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" event={"ID":"cb68ab36-8c03-4ddc-b363-12f65361b836","Type":"ContainerStarted","Data":"7ed738cee9cf71b2d98ec2b03bf53e1d8cf5a8685bdae541d2fc3619ed0ea344"} Nov 02 13:25:15.448095 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:15.448065 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:15.707626 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:15.707545 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:16.250666 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:16.250636 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" event={"ID":"cb68ab36-8c03-4ddc-b363-12f65361b836","Type":"ContainerStarted","Data":"3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98"} Nov 02 13:25:16.251047 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:16.250697 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" Nov 02 13:25:16.367603 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:16.367558 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" podStartSLOduration=2.36754672 podStartE2EDuration="2.36754672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:25:16.36582945 +0000 UTC m=+5333.396065701" watchObservedRunningTime="2026-11-02 13:25:16.36754672 +0000 UTC m=+5333.397782984" Nov 02 13:25:18.191033 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:18.191001 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:21.251143 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:21.251098 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" podUID="cb68ab36-8c03-4ddc-b363-12f65361b836" containerName="payload-pre-processing" containerID="cri-o://3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98" gracePeriod=25 Nov 02 13:25:21.506622 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:21.506569 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" Nov 02 13:25:21.592693 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:21.592665 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb68ab36-8c03-4ddc-b363-12f65361b836-kube-api-access-5fws8\" (UniqueName: \"kubernetes.io/projected/cb68ab36-8c03-4ddc-b363-12f65361b836-kube-api-access-5fws8\") pod \"cb68ab36-8c03-4ddc-b363-12f65361b836\" (UID: \"cb68ab36-8c03-4ddc-b363-12f65361b836\") " Nov 02 13:25:21.592843 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:21.592735 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb68ab36-8c03-4ddc-b363-12f65361b836-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb68ab36-8c03-4ddc-b363-12f65361b836-plugins-config-volume\") pod \"cb68ab36-8c03-4ddc-b363-12f65361b836\" (UID: \"cb68ab36-8c03-4ddc-b363-12f65361b836\") " Nov 02 13:25:21.593078 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:21.593054 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb68ab36-8c03-4ddc-b363-12f65361b836-plugins-config-volume" pod "cb68ab36-8c03-4ddc-b363-12f65361b836" (UID: "cb68ab36-8c03-4ddc-b363-12f65361b836"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:25:21.594845 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:21.594822 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb68ab36-8c03-4ddc-b363-12f65361b836-kube-api-access-5fws8" pod "cb68ab36-8c03-4ddc-b363-12f65361b836" (UID: "cb68ab36-8c03-4ddc-b363-12f65361b836"). InnerVolumeSpecName "kube-api-access-5fws8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:25:21.694229 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:21.694203 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fws8\" (UniqueName: \"kubernetes.io/projected/cb68ab36-8c03-4ddc-b363-12f65361b836-kube-api-access-5fws8\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:25:21.694229 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:21.694227 4312 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb68ab36-8c03-4ddc-b363-12f65361b836-plugins-config-volume\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:25:22.276986 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:22.276956 4312 generic.go:356] "Generic (PLEG): container finished" podID="cb68ab36-8c03-4ddc-b363-12f65361b836" containerID="3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98" exitCode=0 Nov 02 13:25:22.277432 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:22.277004 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" Nov 02 13:25:22.277432 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:22.277050 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" event={"ID":"cb68ab36-8c03-4ddc-b363-12f65361b836","Type":"ContainerDied","Data":"3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98"} Nov 02 13:25:22.277432 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:22.277102 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6" event={"ID":"cb68ab36-8c03-4ddc-b363-12f65361b836","Type":"ContainerDied","Data":"7ed738cee9cf71b2d98ec2b03bf53e1d8cf5a8685bdae541d2fc3619ed0ea344"} Nov 02 13:25:22.277432 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:22.277124 4312 scope.go:122] "RemoveContainer" containerID="3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98" Nov 02 13:25:22.286338 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:22.286318 4312 scope.go:122] "RemoveContainer" containerID="3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98" Nov 02 13:25:22.286634 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:25:22.286615 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98\": container with ID starting with 3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98 not found: ID does not exist" containerID="3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98" Nov 02 13:25:22.286698 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:22.286642 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98"} err="failed to get container status \"3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98\": rpc error: code = NotFound desc = could not find container \"3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98\": container with ID starting with 3cf7f3ab4a55a6eb0ec7af3cdf00c2304e303d6446e79f05dd49e28337435d98 not found: ID does not exist" Nov 02 13:25:22.440486 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:22.440464 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6"] Nov 02 13:25:22.489982 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:22.489958 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-193883-748669b78c-gsnv6"] Nov 02 13:25:23.075006 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:23.074978 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb68ab36-8c03-4ddc-b363-12f65361b836" path="/var/lib/kubelet/pods/cb68ab36-8c03-4ddc-b363-12f65361b836/volumes" Nov 02 13:25:24.322803 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:24.322732 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:24.917461 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:24.917426 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:25.164711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:25.164681 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:25.413289 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:25.413257 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:28.727399 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:28.727366 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:29.234829 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:29.234792 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:29.375357 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:25:29.375321 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:25:31.902640 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:31.902605 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:34.080831 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:34.080786 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:37.006743 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:37.006658 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:43.808699 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:43.808664 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:50.396753 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:50.396713 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:51.155897 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:51.155862 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:55.802910 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:25:55.802870 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:25:59.373918 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:25:59.373885 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:26:02.506607 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:02.506571 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:03.222981 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.222949 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x"] Nov 02 13:26:03.223306 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.223292 4312 cpu_manager.go:414] "containerMap: removing container" podUID="cb68ab36-8c03-4ddc-b363-12f65361b836" containerName="payload-pre-processing" Nov 02 13:26:03.223354 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.223307 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb68ab36-8c03-4ddc-b363-12f65361b836" containerName="payload-pre-processing" Nov 02 13:26:03.223390 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.223370 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb68ab36-8c03-4ddc-b363-12f65361b836" containerName="payload-pre-processing" Nov 02 13:26:03.226135 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.226120 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.327631 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.327596 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1e6e34\"/\"kube-root-ca.crt\"" Nov 02 13:26:03.327815 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.327642 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1e6e34\"/\"openshift-service-ca.crt\"" Nov 02 13:26:03.327815 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.327704 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1e6e34\"/\"ipp-route-1e6e34-kserve-self-signed-certs\"" Nov 02 13:26:03.327815 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.327793 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1e6e34\"/\"default-dockercfg-2glft\"" Nov 02 13:26:03.329545 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.329525 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x"] Nov 02 13:26:03.340585 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.340563 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a965191-a40b-4e43-9661-0ca31edb47ff-tls-certs\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.340694 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.340592 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-model-cache\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.340694 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.340610 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-dshm\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.340772 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.340710 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-home\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.340772 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.340736 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-tmp-dir\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.340772 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.340760 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m2jd\" (UniqueName: \"kubernetes.io/projected/6a965191-a40b-4e43-9661-0ca31edb47ff-kube-api-access-9m2jd\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.412725 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.412692 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:03.441527 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.441484 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a965191-a40b-4e43-9661-0ca31edb47ff-tls-certs\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.441714 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.441532 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-model-cache\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.441714 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.441552 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-dshm\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.441714 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.441594 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-home\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.441714 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.441615 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-tmp-dir\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.441968 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.441944 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-model-cache\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.442046 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.441994 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m2jd\" (UniqueName: \"kubernetes.io/projected/6a965191-a40b-4e43-9661-0ca31edb47ff-kube-api-access-9m2jd\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.442046 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.442022 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-home\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.442356 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.442313 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-tmp-dir\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.444077 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.444052 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-dshm\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.444371 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.444351 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a965191-a40b-4e43-9661-0ca31edb47ff-tls-certs\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.539074 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.539011 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m2jd\" (UniqueName: \"kubernetes.io/projected/6a965191-a40b-4e43-9661-0ca31edb47ff-kube-api-access-9m2jd\") pod \"ipp-route-1e6e34-kserve-77c77f96df-zbr4x\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:03.836961 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:03.836878 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:04.208965 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:04.208941 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x"] Nov 02 13:26:04.211204 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:26:04.211172 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a965191_a40b_4e43_9661_0ca31edb47ff.slice/crio-6cfbf260a02a14ce2c147e90663d06822e58679923e95963aafaeb8ecadfd88f WatchSource:0}: Error finding container 6cfbf260a02a14ce2c147e90663d06822e58679923e95963aafaeb8ecadfd88f: Status 404 returned error can't find the container with id 6cfbf260a02a14ce2c147e90663d06822e58679923e95963aafaeb8ecadfd88f Nov 02 13:26:04.483938 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:04.483846 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" event={"ID":"6a965191-a40b-4e43-9661-0ca31edb47ff","Type":"ContainerStarted","Data":"c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca"} Nov 02 13:26:04.483938 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:04.483897 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" event={"ID":"6a965191-a40b-4e43-9661-0ca31edb47ff","Type":"ContainerStarted","Data":"6cfbf260a02a14ce2c147e90663d06822e58679923e95963aafaeb8ecadfd88f"} Nov 02 13:26:04.603210 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:04.603160 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" podStartSLOduration=1.603143827 podStartE2EDuration="1.603143827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:26:04.603019878 +0000 UTC m=+5381.633256129" watchObservedRunningTime="2026-11-02 13:26:04.603143827 +0000 UTC m=+5381.633380078" Nov 02 13:26:04.837390 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:04.837312 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:09.096377 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:09.096334 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:13.837472 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:13.837426 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:13.850084 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:13.850053 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:14.204377 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:14.204342 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:14.451915 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:14.451882 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:14.540190 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:14.540122 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:26:20.900140 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:20.900088 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:27.514063 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:27.514028 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:29.373838 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:26:29.373800 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:26:32.602970 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:32.602941 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:34.425901 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:34.425857 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:39.402514 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:39.402470 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:46.002196 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:46.002143 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:51.111389 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:51.111359 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:57.879927 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:26:57.879883 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:26:59.374111 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:26:59.374075 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:27:01.282586 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:01.282548 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:04.493121 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:04.493083 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:06.061290 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:06.061209 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x"] Nov 02 13:27:06.254848 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:06.254813 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:09.605969 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:09.605921 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:16.523666 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:16.523630 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:21.080443 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.080405 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" podUID="6a965191-a40b-4e43-9661-0ca31edb47ff" containerName="main" containerID="cri-o://c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca" gracePeriod=45 Nov 02 13:27:21.324109 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.324084 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:27:21.462988 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.462959 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-home\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-home\") pod \"6a965191-a40b-4e43-9661-0ca31edb47ff\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " Nov 02 13:27:21.463158 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.463024 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-tmp-dir\") pod \"6a965191-a40b-4e43-9661-0ca31edb47ff\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " Nov 02 13:27:21.463158 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.463071 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-dshm\") pod \"6a965191-a40b-4e43-9661-0ca31edb47ff\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " Nov 02 13:27:21.463158 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.463103 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a965191-a40b-4e43-9661-0ca31edb47ff-kube-api-access-9m2jd\" (UniqueName: \"kubernetes.io/projected/6a965191-a40b-4e43-9661-0ca31edb47ff-kube-api-access-9m2jd\") pod \"6a965191-a40b-4e43-9661-0ca31edb47ff\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " Nov 02 13:27:21.463158 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.463153 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a965191-a40b-4e43-9661-0ca31edb47ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a965191-a40b-4e43-9661-0ca31edb47ff-tls-certs\") pod \"6a965191-a40b-4e43-9661-0ca31edb47ff\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " Nov 02 13:27:21.463399 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.463197 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-model-cache\") pod \"6a965191-a40b-4e43-9661-0ca31edb47ff\" (UID: \"6a965191-a40b-4e43-9661-0ca31edb47ff\") " Nov 02 13:27:21.463399 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.463230 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-home" pod "6a965191-a40b-4e43-9661-0ca31edb47ff" (UID: "6a965191-a40b-4e43-9661-0ca31edb47ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:27:21.463399 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.463302 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-tmp-dir" pod "6a965191-a40b-4e43-9661-0ca31edb47ff" (UID: "6a965191-a40b-4e43-9661-0ca31edb47ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:27:21.464006 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.463515 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-model-cache" pod "6a965191-a40b-4e43-9661-0ca31edb47ff" (UID: "6a965191-a40b-4e43-9661-0ca31edb47ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:27:21.464006 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.463530 4312 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-home\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:27:21.464006 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.463549 4312 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-tmp-dir\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:27:21.465564 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.465531 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a965191-a40b-4e43-9661-0ca31edb47ff-tls-certs" pod "6a965191-a40b-4e43-9661-0ca31edb47ff" (UID: "6a965191-a40b-4e43-9661-0ca31edb47ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:27:21.465741 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.465717 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-dshm" pod "6a965191-a40b-4e43-9661-0ca31edb47ff" (UID: "6a965191-a40b-4e43-9661-0ca31edb47ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:27:21.465882 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.465860 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a965191-a40b-4e43-9661-0ca31edb47ff-kube-api-access-9m2jd" pod "6a965191-a40b-4e43-9661-0ca31edb47ff" (UID: "6a965191-a40b-4e43-9661-0ca31edb47ff"). InnerVolumeSpecName "kube-api-access-9m2jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:27:21.564774 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.564746 4312 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a965191-a40b-4e43-9661-0ca31edb47ff-tls-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:27:21.564774 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.564771 4312 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-model-cache\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:27:21.564936 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.564782 4312 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a965191-a40b-4e43-9661-0ca31edb47ff-dshm\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:27:21.564936 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.564791 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m2jd\" (UniqueName: \"kubernetes.io/projected/6a965191-a40b-4e43-9661-0ca31edb47ff-kube-api-access-9m2jd\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:27:21.840801 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.840726 4312 generic.go:356] "Generic (PLEG): container finished" podID="6a965191-a40b-4e43-9661-0ca31edb47ff" containerID="c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca" exitCode=0 Nov 02 13:27:21.840801 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.840772 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" Nov 02 13:27:21.841127 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.840817 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" event={"ID":"6a965191-a40b-4e43-9661-0ca31edb47ff","Type":"ContainerDied","Data":"c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca"} Nov 02 13:27:21.841127 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.840865 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x" event={"ID":"6a965191-a40b-4e43-9661-0ca31edb47ff","Type":"ContainerDied","Data":"6cfbf260a02a14ce2c147e90663d06822e58679923e95963aafaeb8ecadfd88f"} Nov 02 13:27:21.841127 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.840889 4312 scope.go:122] "RemoveContainer" containerID="c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca" Nov 02 13:27:21.849278 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.849258 4312 scope.go:122] "RemoveContainer" containerID="c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca" Nov 02 13:27:21.849580 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:27:21.849561 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca\": container with ID starting with c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca not found: ID does not exist" containerID="c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca" Nov 02 13:27:21.849637 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:21.849589 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca"} err="failed to get container status \"c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca\": rpc error: code = NotFound desc = could not find container \"c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca\": container with ID starting with c289d9ad6a383db1283b19d500215701bcad2d0afd001e9bb1e9b0240fb4bfca not found: ID does not exist" Nov 02 13:27:22.004701 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:22.004678 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x"] Nov 02 13:27:22.053994 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:22.053969 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-1e6e34/ipp-route-1e6e34-kserve-77c77f96df-zbr4x"] Nov 02 13:27:23.074707 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:23.074676 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a965191-a40b-4e43-9661-0ca31edb47ff" path="/var/lib/kubelet/pods/6a965191-a40b-4e43-9661-0ca31edb47ff/volumes" Nov 02 13:27:23.246407 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:23.246377 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:28.315869 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:28.315833 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:29.373511 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:27:29.373454 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:27:34.823137 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:34.823098 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:36.427986 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:36.427947 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:41.602914 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:41.602874 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:42.097777 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:42.097744 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:42.344626 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:42.344596 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:43.394128 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:43.394092 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:44.155689 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:44.155648 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:47.234691 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:47.234650 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:50.341133 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:50.341091 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:53.814543 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:53.814511 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:56.597145 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:56.597109 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:58.931989 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:27:58.931953 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:27:59.374003 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:27:59.373915 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:28:04.884780 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:04.884696 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:05.428939 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:05.428899 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:05.684700 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:05.684620 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:06.756766 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:06.756738 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:11.406993 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:11.406955 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:11.743667 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:11.743637 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:12.039430 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:12.039350 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:13.117341 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:13.117304 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:13.764229 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:13.764196 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:19.098157 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:19.098119 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:19.462220 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:19.462190 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:24.206699 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:24.206668 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:25.372522 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:25.372468 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:29.373533 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:28:29.373479 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:28:31.101922 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:31.101888 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:35.111760 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:35.111721 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:35.429483 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:35.429423 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:35.937992 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:35.937960 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:36.187779 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:36.187746 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:40.337039 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:40.337002 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:40.586549 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:40.586516 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:41.893524 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:41.893476 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:42.424126 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:28:42.424068 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:28:59.373772 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:28:59.373728 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:29:29.150642 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:29.150604 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:29:29.372861 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:29:29.372825 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:29:29.592219 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:29.592189 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:29:32.297841 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:32.297808 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:29:35.478580 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:35.478541 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:29:37.153910 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:37.153574 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d9wv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:29:37.153910 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:37.153636 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:29:47.251176 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:47.251145 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95"] Nov 02 13:29:47.251628 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:47.251331 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" containerID="cri-o://3ce2099be6094c1300b462afbe8b9cdb464acfe027a9120ccf55125375514829" gracePeriod=30 Nov 02 13:29:47.300534 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:47.300492 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6"] Nov 02 13:29:47.300711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:47.300681 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" containerID="cri-o://3147ebcab222d66dd839f4f503636c92c210d2fc084cee8ea569066f3b7c350b" gracePeriod=30 Nov 02 13:29:51.764289 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:51.764254 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:29:52.514385 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.514348 4312 generic.go:356] "Generic (PLEG): container finished" podID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerID="3ce2099be6094c1300b462afbe8b9cdb464acfe027a9120ccf55125375514829" exitCode=0 Nov 02 13:29:52.514566 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.514417 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" event={"ID":"3fc38d7e-c396-43ac-83dc-76fa63725a08","Type":"ContainerDied","Data":"3ce2099be6094c1300b462afbe8b9cdb464acfe027a9120ccf55125375514829"} Nov 02 13:29:52.516168 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.516148 4312 generic.go:356] "Generic (PLEG): container finished" podID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerID="3147ebcab222d66dd839f4f503636c92c210d2fc084cee8ea569066f3b7c350b" exitCode=0 Nov 02 13:29:52.516265 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.516205 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" event={"ID":"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8","Type":"ContainerDied","Data":"3147ebcab222d66dd839f4f503636c92c210d2fc084cee8ea569066f3b7c350b"} Nov 02 13:29:52.539570 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.539480 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:29:52.562628 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.562607 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:29:52.644676 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.644650 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-credential-socket\") pod \"3fc38d7e-c396-43ac-83dc-76fa63725a08\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " Nov 02 13:29:52.644783 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.644684 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-envoy\") pod \"3fc38d7e-c396-43ac-83dc-76fa63725a08\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " Nov 02 13:29:52.644783 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.644701 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-data\") pod \"3fc38d7e-c396-43ac-83dc-76fa63725a08\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " Nov 02 13:29:52.644783 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.644739 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-data\") pod \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " Nov 02 13:29:52.644783 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.644766 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-credential-socket\") pod \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " Nov 02 13:29:52.644996 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.644784 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-kube-api-access-sf62p\" (UniqueName: \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-kube-api-access-sf62p\") pod \"3fc38d7e-c396-43ac-83dc-76fa63725a08\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " Nov 02 13:29:52.644996 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.644809 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-envoy\") pod \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " Nov 02 13:29:52.644996 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.644844 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-socket\") pod \"3fc38d7e-c396-43ac-83dc-76fa63725a08\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " Nov 02 13:29:52.645162 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.644958 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-credential-socket" pod "3fc38d7e-c396-43ac-83dc-76fa63725a08" (UID: "3fc38d7e-c396-43ac-83dc-76fa63725a08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:29:52.645162 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645039 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-data" pod "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" (UID: "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:29:52.645162 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645042 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-socket\") pod \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " Nov 02 13:29:52.645162 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645057 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-credential-socket" pod "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" (UID: "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:29:52.645162 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645087 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-certs\") pod \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " Nov 02 13:29:52.645162 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645131 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc38d7e-c396-43ac-83dc-76fa63725a08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc38d7e-c396-43ac-83dc-76fa63725a08-istiod-ca-cert\") pod \"3fc38d7e-c396-43ac-83dc-76fa63725a08\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645202 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-token\" (UniqueName: \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-token\") pod \"3fc38d7e-c396-43ac-83dc-76fa63725a08\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645213 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-data" pod "3fc38d7e-c396-43ac-83dc-76fa63725a08" (UID: "3fc38d7e-c396-43ac-83dc-76fa63725a08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645227 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-socket" pod "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" (UID: "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645233 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istiod-ca-cert\") pod \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645262 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-podinfo\") pod \"3fc38d7e-c396-43ac-83dc-76fa63725a08\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645296 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-certs\") pod \"3fc38d7e-c396-43ac-83dc-76fa63725a08\" (UID: \"3fc38d7e-c396-43ac-83dc-76fa63725a08\") " Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645343 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-podinfo\") pod \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645381 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-token\" (UniqueName: \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-token\") pod \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645382 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-certs" pod "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" (UID: "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645435 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-kube-api-access-gpjpr\" (UniqueName: \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-kube-api-access-gpjpr\") pod \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\" (UID: \"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8\") " Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645582 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istiod-ca-cert" pod "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" (UID: "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:29:52.645711 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645620 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-certs" pod "3fc38d7e-c396-43ac-83dc-76fa63725a08" (UID: "3fc38d7e-c396-43ac-83dc-76fa63725a08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:29:52.646410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645826 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.646410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645845 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.646410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645861 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.646410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645875 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.646410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645889 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.646410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645903 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.646410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645917 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.646410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645930 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.646410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.645581 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc38d7e-c396-43ac-83dc-76fa63725a08-istiod-ca-cert" pod "3fc38d7e-c396-43ac-83dc-76fa63725a08" (UID: "3fc38d7e-c396-43ac-83dc-76fa63725a08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:29:52.646410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.646044 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-socket" pod "3fc38d7e-c396-43ac-83dc-76fa63725a08" (UID: "3fc38d7e-c396-43ac-83dc-76fa63725a08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:29:52.647430 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.647407 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-envoy" pod "3fc38d7e-c396-43ac-83dc-76fa63725a08" (UID: "3fc38d7e-c396-43ac-83dc-76fa63725a08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:29:52.647557 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.647535 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-kube-api-access-sf62p" pod "3fc38d7e-c396-43ac-83dc-76fa63725a08" (UID: "3fc38d7e-c396-43ac-83dc-76fa63725a08"). InnerVolumeSpecName "kube-api-access-sf62p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:29:52.647811 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.647783 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-envoy" pod "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" (UID: "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:29:52.647946 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.647924 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-kube-api-access-gpjpr" pod "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" (UID: "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8"). InnerVolumeSpecName "kube-api-access-gpjpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:29:52.648222 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.648194 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-token" pod "3fc38d7e-c396-43ac-83dc-76fa63725a08" (UID: "3fc38d7e-c396-43ac-83dc-76fa63725a08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:29:52.648313 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.648296 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-podinfo" pod "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" (UID: "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 13:29:52.648362 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.648347 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-podinfo" pod "3fc38d7e-c396-43ac-83dc-76fa63725a08" (UID: "3fc38d7e-c396-43ac-83dc-76fa63725a08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 13:29:52.648772 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.648754 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-token" pod "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" (UID: "6c0fa5df-699a-4f85-a72d-bf6fb8836fb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:29:52.747335 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.747304 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.747335 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.747326 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.747335 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.747336 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.747592 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.747344 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.747592 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.747353 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpjpr\" (UniqueName: \"kubernetes.io/projected/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-kube-api-access-gpjpr\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.747592 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.747363 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.747592 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.747372 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf62p\" (UniqueName: \"kubernetes.io/projected/3fc38d7e-c396-43ac-83dc-76fa63725a08-kube-api-access-sf62p\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.747592 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.747381 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.747592 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.747389 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc38d7e-c396-43ac-83dc-76fa63725a08-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:52.747592 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:52.747397 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc38d7e-c396-43ac-83dc-76fa63725a08-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:29:53.522279 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:53.522248 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" Nov 02 13:29:53.522279 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:53.522265 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95" event={"ID":"3fc38d7e-c396-43ac-83dc-76fa63725a08","Type":"ContainerDied","Data":"ea171cb93e557d3bf89e1fe65252a1d6f780ba3c4877113ec28219d47ed6b362"} Nov 02 13:29:53.522779 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:53.522330 4312 scope.go:122] "RemoveContainer" containerID="3ce2099be6094c1300b462afbe8b9cdb464acfe027a9120ccf55125375514829" Nov 02 13:29:53.523759 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:53.523741 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" Nov 02 13:29:53.523831 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:53.523751 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6" event={"ID":"6c0fa5df-699a-4f85-a72d-bf6fb8836fb8","Type":"ContainerDied","Data":"f00e38c41c9378a112b51ad0de5e6fac043d03ce878449c2743ea472ef947c59"} Nov 02 13:29:53.531163 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:53.531144 4312 scope.go:122] "RemoveContainer" containerID="3147ebcab222d66dd839f4f503636c92c210d2fc084cee8ea569066f3b7c350b" Nov 02 13:29:53.736871 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:53.736843 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6"] Nov 02 13:29:53.786374 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:53.786313 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-tbpd6"] Nov 02 13:29:53.853517 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:53.853448 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95"] Nov 02 13:29:53.859320 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:53.859294 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-8gm95"] Nov 02 13:29:55.074298 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:55.074263 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" path="/var/lib/kubelet/pods/3fc38d7e-c396-43ac-83dc-76fa63725a08/volumes" Nov 02 13:29:55.074692 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:29:55.074681 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" path="/var/lib/kubelet/pods/6c0fa5df-699a-4f85-a72d-bf6fb8836fb8/volumes" Nov 02 13:29:59.375475 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:29:59.375436 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:30:01.016054 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:01.016014 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:30:02.275461 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.275425 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm"] Nov 02 13:30:02.275899 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.275777 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" Nov 02 13:30:02.275899 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.275789 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" Nov 02 13:30:02.275899 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.275801 4312 cpu_manager.go:414] "containerMap: removing container" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" Nov 02 13:30:02.275899 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.275807 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" Nov 02 13:30:02.275899 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.275831 4312 cpu_manager.go:414] "containerMap: removing container" podUID="6a965191-a40b-4e43-9661-0ca31edb47ff" containerName="main" Nov 02 13:30:02.275899 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.275836 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a965191-a40b-4e43-9661-0ca31edb47ff" containerName="main" Nov 02 13:30:02.275899 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.275902 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a965191-a40b-4e43-9661-0ca31edb47ff" containerName="main" Nov 02 13:30:02.276124 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.275913 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c0fa5df-699a-4f85-a72d-bf6fb8836fb8" containerName="istio-proxy" Nov 02 13:30:02.276124 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.275920 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc38d7e-c396-43ac-83dc-76fa63725a08" containerName="istio-proxy" Nov 02 13:30:02.280631 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.280610 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.423055 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.423020 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.423055 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.423058 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.423286 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.423076 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.423286 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.423212 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc3db9f-290e-44ea-b587-0edcae3949ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.423286 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.423242 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.423459 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.423351 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.423459 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.423392 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.423601 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.423468 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.423601 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.423536 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmfzc\" (UniqueName: \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-kube-api-access-pmfzc\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.485871 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.485843 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm"] Nov 02 13:30:02.524536 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.524486 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.524659 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.524542 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.524659 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.524559 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.524659 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.524589 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmfzc\" (UniqueName: \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-kube-api-access-pmfzc\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.524659 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.524612 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.524659 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.524633 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.524659 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.524648 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.524987 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.524678 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc3db9f-290e-44ea-b587-0edcae3949ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.524987 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.524708 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.524987 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.524929 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.525141 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.525075 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.525141 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.525118 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.525227 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.525202 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.525433 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.525416 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc3db9f-290e-44ea-b587-0edcae3949ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.526955 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.526906 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.527041 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.527002 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.585598 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.585574 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmfzc\" (UniqueName: \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-kube-api-access-pmfzc\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.585703 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.585648 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-l2zvm\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:02.593019 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:02.592996 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:03.016559 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:03.016529 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm"] Nov 02 13:30:03.019700 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:30:03.019675 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc3db9f_290e_44ea_b587_0edcae3949ca.slice/crio-5d92cf5a4e999c94a8462e320e91f524eca67d906d1a3a045ac7d63bda5d6cde WatchSource:0}: Error finding container 5d92cf5a4e999c94a8462e320e91f524eca67d906d1a3a045ac7d63bda5d6cde: Status 404 returned error can't find the container with id 5d92cf5a4e999c94a8462e320e91f524eca67d906d1a3a045ac7d63bda5d6cde Nov 02 13:30:03.021760 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:03.021726 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:30:03.021830 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:03.021791 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:30:03.021830 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:03.021821 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:30:03.577895 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:03.577861 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" event={"ID":"acc3db9f-290e-44ea-b587-0edcae3949ca","Type":"ContainerStarted","Data":"6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2"} Nov 02 13:30:03.578396 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:03.577917 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" event={"ID":"acc3db9f-290e-44ea-b587-0edcae3949ca","Type":"ContainerStarted","Data":"5d92cf5a4e999c94a8462e320e91f524eca67d906d1a3a045ac7d63bda5d6cde"} Nov 02 13:30:03.593864 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:03.593843 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:04.594879 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:04.594848 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-l2zvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 13:30:04.595274 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:04.594904 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 13:30:05.593802 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:05.593722 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-l2zvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 13:30:05.593802 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:05.593768 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 13:30:06.593862 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:06.593822 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-l2zvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 13:30:06.594256 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:06.593887 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 13:30:07.593720 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:07.593690 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-l2zvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 13:30:07.593895 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:07.593738 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 13:30:08.593999 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:08.593964 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-l2zvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 13:30:08.594396 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:08.594015 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 13:30:09.593573 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:09.593543 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-l2zvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 13:30:09.593747 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:09.593600 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 13:30:10.593666 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:10.593629 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-l2zvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 13:30:10.594062 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:10.593679 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 13:30:12.270310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:12.270275 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:12.270798 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:12.270669 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:12.271440 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:12.271412 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:30:12.440459 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:12.440411 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podStartSLOduration=10.440399785 podStartE2EDuration="10.440399785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:03.756836992 +0000 UTC m=+5620.787073243" watchObservedRunningTime="2026-11-02 13:30:12.440399785 +0000 UTC m=+5629.470636035" Nov 02 13:30:17.320244 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.320214 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c"] Nov 02 13:30:17.323453 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.323438 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.368117 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.368089 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6km\" (UniqueName: \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-kube-api-access-hb6km\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.368226 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.368119 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.368226 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.368142 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.368226 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.368198 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.368382 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.368344 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.368382 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.368363 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.368484 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.368438 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.368484 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.368473 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.368587 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.368526 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.430659 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.430636 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c"] Nov 02 13:30:17.469471 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.469440 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6km\" (UniqueName: \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-kube-api-access-hb6km\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.469604 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.469478 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.469604 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.469528 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.469604 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.469569 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.469878 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.469851 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.469992 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.469907 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.469992 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.469946 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.469992 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.469969 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.470173 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.470007 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.470173 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.470052 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.470173 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.470097 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.470315 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.470296 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.470650 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.470634 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.470705 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.470635 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.471888 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.471861 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.472234 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.472218 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.530511 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.530474 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.530642 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.530619 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6km\" (UniqueName: \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-kube-api-access-hb6km\") pod \"maas-default-gateway-openshift-default-6dcf757899-t2z9c\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:17.635907 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:17.635830 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:18.061187 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:18.061162 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c"] Nov 02 13:30:18.063213 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:30:18.063188 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d373ab_9fa1_4fae_b884_fba9bf3c401d.slice/crio-b8aceaf7eb95a5ba9cf5e8be242dd8d0c34ddaf064be6213f7b4ecd83f027940 WatchSource:0}: Error finding container b8aceaf7eb95a5ba9cf5e8be242dd8d0c34ddaf064be6213f7b4ecd83f027940: Status 404 returned error can't find the container with id b8aceaf7eb95a5ba9cf5e8be242dd8d0c34ddaf064be6213f7b4ecd83f027940 Nov 02 13:30:18.065212 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:18.065179 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:30:18.065302 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:18.065259 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:30:18.065367 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:18.065303 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:30:18.646641 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:18.646596 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" event={"ID":"97d373ab-9fa1-4fae-b884-fba9bf3c401d","Type":"ContainerStarted","Data":"e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9"} Nov 02 13:30:18.647075 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:18.646679 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" event={"ID":"97d373ab-9fa1-4fae-b884-fba9bf3c401d","Type":"ContainerStarted","Data":"b8aceaf7eb95a5ba9cf5e8be242dd8d0c34ddaf064be6213f7b4ecd83f027940"} Nov 02 13:30:19.636726 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:19.636687 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:19.638192 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:19.638163 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-t2z9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 13:30:19.638317 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:19.638218 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 13:30:20.636805 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:20.636763 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-t2z9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 13:30:20.637286 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:20.636822 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 13:30:21.636909 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:21.636877 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-t2z9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 13:30:21.637320 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:21.636923 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 13:30:22.636985 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:22.636954 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-t2z9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 13:30:22.637375 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:22.637007 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 13:30:23.636707 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:23.636677 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-t2z9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 13:30:23.636886 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:23.636727 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 13:30:24.637146 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:24.637113 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-t2z9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 13:30:24.637595 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:24.637166 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 13:30:25.636293 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:25.636258 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-t2z9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Nov 02 13:30:25.636470 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:25.636308 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Nov 02 13:30:27.290752 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:27.290718 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:27.291247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:27.291124 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:27.292028 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:27.292002 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:30:27.462209 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:27.462159 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" podStartSLOduration=10.46214648 podStartE2EDuration="10.46214648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:18.770784932 +0000 UTC m=+5635.801021182" watchObservedRunningTime="2026-11-02 13:30:27.46214648 +0000 UTC m=+5644.492382786" Nov 02 13:30:29.372173 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:30:29.372132 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:30:35.824802 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:30:35.824768 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:30:59.374768 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:30:59.374727 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:31:20.173850 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:20.173810 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:31:20.520225 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:20.520153 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:31:20.828842 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:20.828762 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:31:28.311684 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:28.311621 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-l2zvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:31:28.312249 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:28.311681 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:31:28.333635 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:28.333610 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-t2z9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:31:28.333778 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:28.333650 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:31:29.372644 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:31:29.372610 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:31:36.333773 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:36.333691 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:31:36.608571 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:36.605155 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:31:37.146631 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:37.146598 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d9wv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:31:37.146814 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:37.146651 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:31:41.609482 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:41.609409 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:31:41.864890 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:41.864821 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:31:59.373982 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:31:59.373940 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:31:59.413346 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:59.413319 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:31:59.716602 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:31:59.711098 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:32:29.373719 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:32:29.373676 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:32:36.738247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:32:36.738183 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:32:51.276926 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:32:51.276889 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:32:59.374467 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:32:59.374416 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:33:13.428705 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:33:13.428611 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:33:29.431687 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:33:29.431649 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:33:37.058082 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:33:37.058037 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:33:59.374146 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:33:59.374101 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:34:29.372607 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:34:29.372571 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:34:37.379903 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:34:37.379788 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:34:59.372991 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:34:59.372951 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:35:29.374441 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:35:29.374400 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:35:37.719172 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:35:37.719098 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:35:59.372858 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:35:59.372819 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:36:29.372861 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:36:29.372824 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:36:38.064146 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:36:38.064063 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:36:53.200938 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:36:53.200904 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:36:53.846638 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:36:53.846605 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:36:54.363107 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:36:54.362963 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:36:58.075034 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:36:58.074945 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:36:58.426130 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:36:58.426082 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:36:59.372300 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:36:59.372262 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:37:01.376207 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:01.376175 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:01.619718 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:01.619671 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:04.632660 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:04.632628 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:04.912148 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:04.912119 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:10.036719 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:10.036683 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:14.739284 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:14.739252 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:19.676941 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:19.676900 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:22.796115 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:22.796069 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:23.329752 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:23.329720 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:23.594426 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:23.594335 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:25.045778 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:25.045748 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:26.223033 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:26.222996 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:29.371922 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:37:29.371885 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:37:30.017178 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:30.017146 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:34.920387 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:34.920357 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:38.457957 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:38.457861 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:39.811852 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:39.811810 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:44.699154 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:44.699118 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:49.740566 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:49.740533 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:54.698850 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:54.698812 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:37:59.373520 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:37:59.373464 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:37:59.720366 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:37:59.720335 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:04.691411 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:04.691378 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:06.650902 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:06.650860 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:09.618083 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:09.618047 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:14.601884 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:14.601848 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:19.499702 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:19.499667 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:20.317591 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:20.317549 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:24.215290 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:24.215246 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:24.476766 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:24.476633 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:24.727555 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:24.727414 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:29.264542 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:29.264485 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:29.374361 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:38:29.374323 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:38:29.519428 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:29.519360 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:30.958845 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:30.958809 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:31.613983 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:31.613953 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:31.882208 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:31.882138 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:34.202074 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:34.202044 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:36.760915 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:36.760886 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:37.016590 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:37.016491 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:38.783752 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:38.783713 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:58.174780 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:58.174735 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:58.421268 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:38:58.421221 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:38:59.371898 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:38:59.371863 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:39:06.085489 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:06.085367 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:06.391590 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:06.391505 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:06.657087 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:06.657011 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:07.358537 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:07.358479 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:08.377146 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:08.377101 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:08.631267 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:08.631170 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:09.205467 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:09.205418 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:09.454923 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:09.454887 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:13.497177 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:13.497141 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:13.751183 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:13.751105 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:18.598812 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:18.598782 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:23.504298 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:23.504262 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:28.424150 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:28.424116 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:29.371720 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:39:29.371682 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:39:33.280694 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:33.280665 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:34.675262 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:34.675232 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:34.923328 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:34.923296 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:37.596620 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:37.596578 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:39.115031 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:39.115003 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:39.363309 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:39.363277 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:39.614361 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:39.614319 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:40.057800 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:40.057749 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:45.271539 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:45.271483 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:39:58.316077 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:58.315584 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-l2zvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:39:58.316077 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:58.315649 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:39:58.337623 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:58.337592 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-t2z9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:39:58.337766 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:39:58.337639 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:39:59.439579 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:39:59.439536 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:40:29.373607 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:40:29.373568 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:40:39.472299 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:40:39.472214 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:40:59.373014 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:40:59.372976 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:41:29.435485 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:41:29.435451 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:41:39.837611 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:41:39.837570 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:41:52.173618 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:41:52.173581 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-d9wv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:41:52.174201 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:41:52.174171 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:41:59.373195 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:41:59.373155 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:42:13.755755 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:13.755649 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:42:14.027633 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:14.027554 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:42:16.448475 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.448446 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6459565456-qf5nj"] Nov 02 13:42:16.451973 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.451956 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.534846 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.534816 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7afc4c-b5cd-4f04-ba9c-e688826371e8-maas-api-tls\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.534997 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.534860 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7afc4c-b5cd-4f04-ba9c-e688826371e8-openshift-service-ca\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.534997 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.534961 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be7afc4c-b5cd-4f04-ba9c-e688826371e8-metrics-certs\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.535151 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.535017 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flpvj\" (UniqueName: \"kubernetes.io/projected/be7afc4c-b5cd-4f04-ba9c-e688826371e8-kube-api-access-flpvj\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.552682 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.552654 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6459565456-qf5nj"] Nov 02 13:42:16.635530 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.635486 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be7afc4c-b5cd-4f04-ba9c-e688826371e8-metrics-certs\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.635644 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.635568 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flpvj\" (UniqueName: \"kubernetes.io/projected/be7afc4c-b5cd-4f04-ba9c-e688826371e8-kube-api-access-flpvj\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.635644 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.635609 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7afc4c-b5cd-4f04-ba9c-e688826371e8-maas-api-tls\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.635754 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.635657 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7afc4c-b5cd-4f04-ba9c-e688826371e8-openshift-service-ca\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.636277 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.636257 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be7afc4c-b5cd-4f04-ba9c-e688826371e8-openshift-service-ca\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.638017 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.637994 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be7afc4c-b5cd-4f04-ba9c-e688826371e8-metrics-certs\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.638105 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.638056 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be7afc4c-b5cd-4f04-ba9c-e688826371e8-maas-api-tls\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.696121 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.696102 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flpvj\" (UniqueName: \"kubernetes.io/projected/be7afc4c-b5cd-4f04-ba9c-e688826371e8-kube-api-access-flpvj\") pod \"maas-api-6459565456-qf5nj\" (UID: \"be7afc4c-b5cd-4f04-ba9c-e688826371e8\") " pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:16.764490 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:16.764431 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:17.141226 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:17.141204 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6459565456-qf5nj"] Nov 02 13:42:17.143448 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:42:17.143418 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7afc4c_b5cd_4f04_ba9c_e688826371e8.slice/crio-46a714ffb8dceee99dc3812bcb37f959af2f0bae0588da6a610eea40560145d0 WatchSource:0}: Error finding container 46a714ffb8dceee99dc3812bcb37f959af2f0bae0588da6a610eea40560145d0: Status 404 returned error can't find the container with id 46a714ffb8dceee99dc3812bcb37f959af2f0bae0588da6a610eea40560145d0 Nov 02 13:42:17.144706 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:17.144690 4312 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 13:42:17.246793 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:17.246768 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" event={"ID":"be7afc4c-b5cd-4f04-ba9c-e688826371e8","Type":"ContainerStarted","Data":"46a714ffb8dceee99dc3812bcb37f959af2f0bae0588da6a610eea40560145d0"} Nov 02 13:42:18.254117 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:18.254070 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" event={"ID":"be7afc4c-b5cd-4f04-ba9c-e688826371e8","Type":"ContainerStarted","Data":"d488c98def34a9337ea16a3f56407d8b5ccb57fec7aa97ff0f64aabf5aa3c30e"} Nov 02 13:42:18.254564 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:18.254202 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:18.377113 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:18.377024 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" podStartSLOduration=1.477770345 podStartE2EDuration="2.377011718s" totalImagesPullingTime="899.241373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:18.375958119 +0000 UTC m=+6355.406194369" watchObservedRunningTime="2026-11-02 13:42:18.377011718 +0000 UTC m=+6355.407247966" Nov 02 13:42:24.265393 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:24.265361 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6459565456-qf5nj" Nov 02 13:42:24.433755 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:24.433724 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd"] Nov 02 13:42:24.434004 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:24.433965 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" podUID="32dda5be-5c7f-41f4-bef6-8eff5d1003fb" containerName="maas-api" containerID="cri-o://b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f" gracePeriod=30 Nov 02 13:42:25.683890 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.683868 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 13:42:25.765440 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.765392 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-kube-api-access-8lr9j\" (UniqueName: \"kubernetes.io/projected/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-kube-api-access-8lr9j\") pod \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " Nov 02 13:42:25.765591 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.765459 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-openshift-service-ca\") pod \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " Nov 02 13:42:25.765591 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.765541 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-metrics-certs\") pod \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " Nov 02 13:42:25.765720 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.765608 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-maas-api-tls\") pod \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\" (UID: \"32dda5be-5c7f-41f4-bef6-8eff5d1003fb\") " Nov 02 13:42:25.765914 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.765879 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-openshift-service-ca" pod "32dda5be-5c7f-41f4-bef6-8eff5d1003fb" (UID: "32dda5be-5c7f-41f4-bef6-8eff5d1003fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:42:25.767640 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.767611 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-maas-api-tls" pod "32dda5be-5c7f-41f4-bef6-8eff5d1003fb" (UID: "32dda5be-5c7f-41f4-bef6-8eff5d1003fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:42:25.767721 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.767637 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-metrics-certs" pod "32dda5be-5c7f-41f4-bef6-8eff5d1003fb" (UID: "32dda5be-5c7f-41f4-bef6-8eff5d1003fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 13:42:25.767721 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.767656 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-kube-api-access-8lr9j" pod "32dda5be-5c7f-41f4-bef6-8eff5d1003fb" (UID: "32dda5be-5c7f-41f4-bef6-8eff5d1003fb"). InnerVolumeSpecName "kube-api-access-8lr9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:42:25.867194 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.867162 4312 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-openshift-service-ca\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:42:25.867194 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.867190 4312 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-metrics-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:42:25.867350 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.867206 4312 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-maas-api-tls\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:42:25.867350 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:25.867220 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lr9j\" (UniqueName: \"kubernetes.io/projected/32dda5be-5c7f-41f4-bef6-8eff5d1003fb-kube-api-access-8lr9j\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:42:26.310175 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:26.310138 4312 generic.go:356] "Generic (PLEG): container finished" podID="32dda5be-5c7f-41f4-bef6-8eff5d1003fb" containerID="b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f" exitCode=0 Nov 02 13:42:26.310405 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:26.310203 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" event={"ID":"32dda5be-5c7f-41f4-bef6-8eff5d1003fb","Type":"ContainerDied","Data":"b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f"} Nov 02 13:42:26.310405 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:26.310207 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" Nov 02 13:42:26.310405 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:26.310244 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd" event={"ID":"32dda5be-5c7f-41f4-bef6-8eff5d1003fb","Type":"ContainerDied","Data":"23dbb72b95c2837dab87a16352ddaa96f3a0a39e5ee8e8f7d52636291b06949a"} Nov 02 13:42:26.310405 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:26.310265 4312 scope.go:122] "RemoveContainer" containerID="b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f" Nov 02 13:42:26.321287 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:26.321179 4312 scope.go:122] "RemoveContainer" containerID="b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f" Nov 02 13:42:26.321621 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:42:26.321598 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f\": container with ID starting with b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f not found: ID does not exist" containerID="b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f" Nov 02 13:42:26.321713 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:26.321628 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f"} err="failed to get container status \"b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f\": rpc error: code = NotFound desc = could not find container \"b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f\": container with ID starting with b4508fa529cfc846783e19854a5a7cb9d08bdb2383a6095274c75bafb06b483f not found: ID does not exist" Nov 02 13:42:26.472713 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:26.472679 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd"] Nov 02 13:42:26.522253 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:26.522226 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6866d668fb-cfdrd"] Nov 02 13:42:27.074478 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:27.074433 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32dda5be-5c7f-41f4-bef6-8eff5d1003fb" path="/var/lib/kubelet/pods/32dda5be-5c7f-41f4-bef6-8eff5d1003fb/volumes" Nov 02 13:42:29.372415 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:42:29.372382 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:42:31.513038 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:31.512999 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:42:37.352849 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:37.352813 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:42:40.219748 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:40.219707 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:42:50.381146 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:42:50.381111 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:42:59.372251 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:42:59.372215 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:43:29.373705 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:43:29.373656 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:43:40.448565 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:43:40.448465 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:43:59.373438 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:43:59.373401 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:44:02.027752 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:44:02.027716 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:02.305079 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:44:02.304999 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:03.102320 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:44:03.102290 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:29.291288 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:44:29.291241 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:29.373286 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:44:29.373249 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:44:35.326339 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:44:35.326299 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:40.880972 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:44:40.880932 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:58.297628 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:44:58.297588 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:44:59.373158 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:44:59.373122 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:45:00.094868 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:45:00.094834 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:29.372386 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:45:29.372345 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:45:41.229840 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:45:41.229809 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:45:59.373795 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:45:59.373755 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:45:59.747038 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:45:59.747007 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:46:00.002897 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:46:00.002828 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:46:16.563489 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:46:16.563458 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:46:22.437563 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:46:22.437522 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:46:29.372749 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:46:29.372712 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:46:41.577168 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:46:41.577086 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:46:59.372137 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:46:59.372100 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:47:29.373354 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:47:29.373315 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:47:41.969078 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:41.969042 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:46.701043 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:46.701005 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:46.963231 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:46.963155 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:51.941205 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:51.941162 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2"] Nov 02 13:47:51.941736 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:51.941427 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" containerID="cri-o://d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259" gracePeriod=30 Nov 02 13:47:51.990607 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:51.990578 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c"] Nov 02 13:47:51.990775 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:51.990756 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" containerID="cri-o://e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9" gracePeriod=30 Nov 02 13:47:51.991929 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:51.991908 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm"] Nov 02 13:47:51.992099 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:51.992080 4312 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" containerID="cri-o://6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2" gracePeriod=30 Nov 02 13:47:57.260784 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.260757 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:47:57.274524 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.274487 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:47:57.280923 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.280905 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:47:57.389954 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.389929 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-kube-api-access-7g8ht\" (UniqueName: \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-kube-api-access-7g8ht\") pod \"8426f988-1fd5-442f-957d-1e1a58590ee0\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " Nov 02 13:47:57.390082 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.389961 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-kube-api-access-pmfzc\" (UniqueName: \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-kube-api-access-pmfzc\") pod \"acc3db9f-290e-44ea-b587-0edcae3949ca\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " Nov 02 13:47:57.390082 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.389993 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-certs\") pod \"8426f988-1fd5-442f-957d-1e1a58590ee0\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " Nov 02 13:47:57.390082 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390021 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-kube-api-access-hb6km\" (UniqueName: \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-kube-api-access-hb6km\") pod \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " Nov 02 13:47:57.390082 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390075 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-certs\") pod \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " Nov 02 13:47:57.390310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390107 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-envoy\") pod \"8426f988-1fd5-442f-957d-1e1a58590ee0\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " Nov 02 13:47:57.390310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390134 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-socket\") pod \"8426f988-1fd5-442f-957d-1e1a58590ee0\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " Nov 02 13:47:57.390310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390161 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-certs\") pod \"acc3db9f-290e-44ea-b587-0edcae3949ca\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " Nov 02 13:47:57.390310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390193 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-token\" (UniqueName: \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-token\") pod \"8426f988-1fd5-442f-957d-1e1a58590ee0\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " Nov 02 13:47:57.390310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390225 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-envoy\") pod \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " Nov 02 13:47:57.390310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390251 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-podinfo\") pod \"acc3db9f-290e-44ea-b587-0edcae3949ca\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " Nov 02 13:47:57.390310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390279 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-socket\") pod \"acc3db9f-290e-44ea-b587-0edcae3949ca\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " Nov 02 13:47:57.390310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390291 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-certs" pod "8426f988-1fd5-442f-957d-1e1a58590ee0" (UID: "8426f988-1fd5-442f-957d-1e1a58590ee0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.390310 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390306 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-data\") pod \"8426f988-1fd5-442f-957d-1e1a58590ee0\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390325 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-certs" pod "97d373ab-9fa1-4fae-b884-fba9bf3c401d" (UID: "97d373ab-9fa1-4fae-b884-fba9bf3c401d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390359 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-token\" (UniqueName: \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-token\") pod \"acc3db9f-290e-44ea-b587-0edcae3949ca\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390397 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-data\") pod \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390428 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8426f988-1fd5-442f-957d-1e1a58590ee0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8426f988-1fd5-442f-957d-1e1a58590ee0-istiod-ca-cert\") pod \"8426f988-1fd5-442f-957d-1e1a58590ee0\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390538 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istiod-ca-cert\") pod \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390549 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-data" pod "8426f988-1fd5-442f-957d-1e1a58590ee0" (UID: "8426f988-1fd5-442f-957d-1e1a58590ee0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390607 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-credential-socket\") pod \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390642 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-podinfo\") pod \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390679 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-data\") pod \"acc3db9f-290e-44ea-b587-0edcae3949ca\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390712 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-socket\") pod \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390715 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-data" pod "97d373ab-9fa1-4fae-b884-fba9bf3c401d" (UID: "97d373ab-9fa1-4fae-b884-fba9bf3c401d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390763 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-credential-socket\") pod \"8426f988-1fd5-442f-957d-1e1a58590ee0\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " Nov 02 13:47:57.390810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390807 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-credential-socket\") pod \"acc3db9f-290e-44ea-b587-0edcae3949ca\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " Nov 02 13:47:57.391514 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390854 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-envoy\") pod \"acc3db9f-290e-44ea-b587-0edcae3949ca\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " Nov 02 13:47:57.391514 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390896 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-podinfo\") pod \"8426f988-1fd5-442f-957d-1e1a58590ee0\" (UID: \"8426f988-1fd5-442f-957d-1e1a58590ee0\") " Nov 02 13:47:57.391514 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390924 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-token\" (UniqueName: \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-token\") pod \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\" (UID: \"97d373ab-9fa1-4fae-b884-fba9bf3c401d\") " Nov 02 13:47:57.391514 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.390956 4312 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc3db9f-290e-44ea-b587-0edcae3949ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc3db9f-290e-44ea-b587-0edcae3949ca-istiod-ca-cert\") pod \"acc3db9f-290e-44ea-b587-0edcae3949ca\" (UID: \"acc3db9f-290e-44ea-b587-0edcae3949ca\") " Nov 02 13:47:57.391514 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391296 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.391514 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391316 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.391514 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391332 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.391514 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391345 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-socket" pod "8426f988-1fd5-442f-957d-1e1a58590ee0" (UID: "8426f988-1fd5-442f-957d-1e1a58590ee0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.391514 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391346 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.391986 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391559 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-socket" pod "acc3db9f-290e-44ea-b587-0edcae3949ca" (UID: "acc3db9f-290e-44ea-b587-0edcae3949ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.391986 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391579 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-certs" pod "acc3db9f-290e-44ea-b587-0edcae3949ca" (UID: "acc3db9f-290e-44ea-b587-0edcae3949ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.391986 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391580 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istiod-ca-cert" pod "97d373ab-9fa1-4fae-b884-fba9bf3c401d" (UID: "97d373ab-9fa1-4fae-b884-fba9bf3c401d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:47:57.391986 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391781 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-credential-socket" pod "97d373ab-9fa1-4fae-b884-fba9bf3c401d" (UID: "97d373ab-9fa1-4fae-b884-fba9bf3c401d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.391986 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391791 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-credential-socket" pod "acc3db9f-290e-44ea-b587-0edcae3949ca" (UID: "acc3db9f-290e-44ea-b587-0edcae3949ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.391986 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391792 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc3db9f-290e-44ea-b587-0edcae3949ca-istiod-ca-cert" pod "acc3db9f-290e-44ea-b587-0edcae3949ca" (UID: "acc3db9f-290e-44ea-b587-0edcae3949ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:47:57.391986 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.391889 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8426f988-1fd5-442f-957d-1e1a58590ee0-istiod-ca-cert" pod "8426f988-1fd5-442f-957d-1e1a58590ee0" (UID: "8426f988-1fd5-442f-957d-1e1a58590ee0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 13:47:57.392384 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.392022 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-data" pod "acc3db9f-290e-44ea-b587-0edcae3949ca" (UID: "acc3db9f-290e-44ea-b587-0edcae3949ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.392384 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.392223 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-socket" pod "97d373ab-9fa1-4fae-b884-fba9bf3c401d" (UID: "97d373ab-9fa1-4fae-b884-fba9bf3c401d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.392384 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.392228 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-credential-socket" pod "8426f988-1fd5-442f-957d-1e1a58590ee0" (UID: "8426f988-1fd5-442f-957d-1e1a58590ee0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.394371 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.394339 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-kube-api-access-hb6km" pod "97d373ab-9fa1-4fae-b884-fba9bf3c401d" (UID: "97d373ab-9fa1-4fae-b884-fba9bf3c401d"). InnerVolumeSpecName "kube-api-access-hb6km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:47:57.394488 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.394427 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-envoy" pod "97d373ab-9fa1-4fae-b884-fba9bf3c401d" (UID: "97d373ab-9fa1-4fae-b884-fba9bf3c401d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.394488 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.394432 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-token" pod "97d373ab-9fa1-4fae-b884-fba9bf3c401d" (UID: "97d373ab-9fa1-4fae-b884-fba9bf3c401d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:47:57.394648 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.394593 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-token" pod "8426f988-1fd5-442f-957d-1e1a58590ee0" (UID: "8426f988-1fd5-442f-957d-1e1a58590ee0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:47:57.394732 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.394709 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-token" pod "acc3db9f-290e-44ea-b587-0edcae3949ca" (UID: "acc3db9f-290e-44ea-b587-0edcae3949ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:47:57.394827 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.394742 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-podinfo" pod "acc3db9f-290e-44ea-b587-0edcae3949ca" (UID: "acc3db9f-290e-44ea-b587-0edcae3949ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 13:47:57.394889 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.394820 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-podinfo" pod "97d373ab-9fa1-4fae-b884-fba9bf3c401d" (UID: "97d373ab-9fa1-4fae-b884-fba9bf3c401d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 13:47:57.394889 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.394865 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-kube-api-access-7g8ht" pod "8426f988-1fd5-442f-957d-1e1a58590ee0" (UID: "8426f988-1fd5-442f-957d-1e1a58590ee0"). InnerVolumeSpecName "kube-api-access-7g8ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:47:57.394889 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.394866 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-kube-api-access-pmfzc" pod "acc3db9f-290e-44ea-b587-0edcae3949ca" (UID: "acc3db9f-290e-44ea-b587-0edcae3949ca"). InnerVolumeSpecName "kube-api-access-pmfzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 13:47:57.394889 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.394870 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-envoy" pod "8426f988-1fd5-442f-957d-1e1a58590ee0" (UID: "8426f988-1fd5-442f-957d-1e1a58590ee0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.395112 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.395036 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-podinfo" pod "8426f988-1fd5-442f-957d-1e1a58590ee0" (UID: "8426f988-1fd5-442f-957d-1e1a58590ee0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 13:47:57.396164 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.396139 4312 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-envoy" pod "acc3db9f-290e-44ea-b587-0edcae3949ca" (UID: "acc3db9f-290e-44ea-b587-0edcae3949ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 13:47:57.492707 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492652 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmfzc\" (UniqueName: \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-kube-api-access-pmfzc\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492707 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492672 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb6km\" (UniqueName: \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-kube-api-access-hb6km\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492707 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492682 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492707 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492690 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492707 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492699 4312 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-certs\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492707 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492707 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492714 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492722 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492730 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492740 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492748 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8426f988-1fd5-442f-957d-1e1a58590ee0-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492757 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492765 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492775 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492783 4312 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-data\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492792 4312 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d373ab-9fa1-4fae-b884-fba9bf3c401d-workload-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492800 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8426f988-1fd5-442f-957d-1e1a58590ee0-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492809 4312 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-credential-socket\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492817 4312 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acc3db9f-290e-44ea-b587-0edcae3949ca-istio-envoy\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492824 4312 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8426f988-1fd5-442f-957d-1e1a58590ee0-istio-podinfo\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492832 4312 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d373ab-9fa1-4fae-b884-fba9bf3c401d-istio-token\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492840 4312 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acc3db9f-290e-44ea-b587-0edcae3949ca-istiod-ca-cert\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.492963 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.492849 4312 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g8ht\" (UniqueName: \"kubernetes.io/projected/8426f988-1fd5-442f-957d-1e1a58590ee0-kube-api-access-7g8ht\") on node \"ip-10-0-141-170.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 13:47:57.743982 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.743914 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:47:57.820548 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.820517 4312 generic.go:356] "Generic (PLEG): container finished" podID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerID="6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2" exitCode=0 Nov 02 13:47:57.820703 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.820575 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" Nov 02 13:47:57.820703 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.820608 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" event={"ID":"acc3db9f-290e-44ea-b587-0edcae3949ca","Type":"ContainerDied","Data":"6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2"} Nov 02 13:47:57.820703 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.820668 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" event={"ID":"acc3db9f-290e-44ea-b587-0edcae3949ca","Type":"ContainerDied","Data":"5d92cf5a4e999c94a8462e320e91f524eca67d906d1a3a045ac7d63bda5d6cde"} Nov 02 13:47:57.820703 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.820700 4312 scope.go:122] "RemoveContainer" containerID="6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2" Nov 02 13:47:57.821913 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.821893 4312 generic.go:356] "Generic (PLEG): container finished" podID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerID="e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9" exitCode=0 Nov 02 13:47:57.822004 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.821941 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" Nov 02 13:47:57.822004 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.821983 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" event={"ID":"97d373ab-9fa1-4fae-b884-fba9bf3c401d","Type":"ContainerDied","Data":"e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9"} Nov 02 13:47:57.822118 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.822038 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c" event={"ID":"97d373ab-9fa1-4fae-b884-fba9bf3c401d","Type":"ContainerDied","Data":"b8aceaf7eb95a5ba9cf5e8be242dd8d0c34ddaf064be6213f7b4ecd83f027940"} Nov 02 13:47:57.823684 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.823660 4312 generic.go:356] "Generic (PLEG): container finished" podID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerID="d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259" exitCode=0 Nov 02 13:47:57.823793 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.823693 4312 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" Nov 02 13:47:57.823793 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.823709 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" event={"ID":"8426f988-1fd5-442f-957d-1e1a58590ee0","Type":"ContainerDied","Data":"d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259"} Nov 02 13:47:57.823793 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.823747 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2" event={"ID":"8426f988-1fd5-442f-957d-1e1a58590ee0","Type":"ContainerDied","Data":"fc0c8de3c47a1ed645edc97283f3c03d29cba0e106e0d56b8fb1b445f8154262"} Nov 02 13:47:57.830210 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.830190 4312 scope.go:122] "RemoveContainer" containerID="6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2" Nov 02 13:47:57.830481 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:47:57.830458 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2\": container with ID starting with 6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2 not found: ID does not exist" containerID="6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2" Nov 02 13:47:57.830624 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.830486 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2"} err="failed to get container status \"6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2\": rpc error: code = NotFound desc = could not find container \"6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2\": container with ID starting with 6cdf489b260d087e289e2ca20698e21a28632fda9b26b5c53b5078918b0242f2 not found: ID does not exist" Nov 02 13:47:57.830624 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.830551 4312 scope.go:122] "RemoveContainer" containerID="e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9" Nov 02 13:47:57.838830 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.838815 4312 scope.go:122] "RemoveContainer" containerID="e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9" Nov 02 13:47:57.839070 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:47:57.839051 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9\": container with ID starting with e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9 not found: ID does not exist" containerID="e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9" Nov 02 13:47:57.839163 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.839077 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9"} err="failed to get container status \"e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9\": rpc error: code = NotFound desc = could not find container \"e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9\": container with ID starting with e6f01ed9ad3b8e052ae69c06db56f931b0e29c4232e1e022bf698605ff247ae9 not found: ID does not exist" Nov 02 13:47:57.839163 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.839092 4312 scope.go:122] "RemoveContainer" containerID="d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259" Nov 02 13:47:57.846559 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.846543 4312 scope.go:122] "RemoveContainer" containerID="d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259" Nov 02 13:47:57.846795 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:47:57.846775 4312 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259\": container with ID starting with d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259 not found: ID does not exist" containerID="d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259" Nov 02 13:47:57.846855 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.846803 4312 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259"} err="failed to get container status \"d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259\": rpc error: code = NotFound desc = could not find container \"d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259\": container with ID starting with d5785025b43c4a0740b8de7c5d2eb2330138a072f63a005cddd5b5d85979c259 not found: ID does not exist" Nov 02 13:47:57.985679 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.985655 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2"] Nov 02 13:47:57.988462 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:57.988442 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-d9wv2"] Nov 02 13:47:58.097187 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:58.097138 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c"] Nov 02 13:47:58.101909 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:58.101885 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-t2z9c"] Nov 02 13:47:58.210413 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:58.210389 4312 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm"] Nov 02 13:47:58.215072 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:58.215050 4312 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm"] Nov 02 13:47:58.270915 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:58.270887 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-l2zvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:47:58.271202 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:58.270948 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-l2zvm" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:47:59.074270 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:59.074235 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" path="/var/lib/kubelet/pods/8426f988-1fd5-442f-957d-1e1a58590ee0/volumes" Nov 02 13:47:59.074677 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:59.074662 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" path="/var/lib/kubelet/pods/97d373ab-9fa1-4fae-b884-fba9bf3c401d/volumes" Nov 02 13:47:59.075046 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:47:59.075034 4312 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" path="/var/lib/kubelet/pods/acc3db9f-290e-44ea-b587-0edcae3949ca/volumes" Nov 02 13:47:59.372422 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:47:59.372347 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:48:17.033813 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:48:17.033699 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:29.372809 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:48:29.372766 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:48:42.259115 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:48:42.259073 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:48:59.373242 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:48:59.373197 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:49:29.374649 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:49:29.374605 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:49:34.249686 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:49:34.249649 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:34.534029 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:49:34.533952 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:35.789464 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:49:35.789383 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:36.039718 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:49:36.039644 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:36.531427 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:49:36.531400 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:43.939974 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:49:43.939935 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:48.574862 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:49:48.574815 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:54.518894 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:49:54.518861 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:49:59.373021 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:49:59.372986 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:50:03.507862 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:03.507832 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:08.067595 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.067556 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj"] Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.067895 4312 cpu_manager.go:414] "containerMap: removing container" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.067907 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.067926 4312 cpu_manager.go:414] "containerMap: removing container" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.067931 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.067941 4312 cpu_manager.go:414] "containerMap: removing container" podUID="32dda5be-5c7f-41f4-bef6-8eff5d1003fb" containerName="maas-api" Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.067946 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32dda5be-5c7f-41f4-bef6-8eff5d1003fb" containerName="maas-api" Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.067962 4312 cpu_manager.go:414] "containerMap: removing container" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.067967 4312 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.068019 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d373ab-9fa1-4fae-b884-fba9bf3c401d" containerName="istio-proxy" Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.068026 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="32dda5be-5c7f-41f4-bef6-8eff5d1003fb" containerName="maas-api" Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.068035 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc3db9f-290e-44ea-b587-0edcae3949ca" containerName="istio-proxy" Nov 02 13:50:08.068098 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.068043 4312 memory_manager.go:365] "RemoveStaleState removing state" podUID="8426f988-1fd5-442f-957d-1e1a58590ee0" containerName="istio-proxy" Nov 02 13:50:08.071128 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.071103 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.134344 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.134320 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 13:50:08.169981 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.169953 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.170097 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.169987 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660bd85f-78bb-418d-aae8-6e888a8f7b31-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.170097 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.170008 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.170198 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.170123 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.170198 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.170163 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.170287 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.170210 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.170287 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.170228 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.170394 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.170287 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.170394 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.170338 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbb8p\" (UniqueName: \"kubernetes.io/projected/660bd85f-78bb-418d-aae8-6e888a8f7b31-kube-api-access-tbb8p\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.194769 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.194747 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c5ldd\"" Nov 02 13:50:08.200273 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.200251 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj"] Nov 02 13:50:08.271478 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.271453 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbb8p\" (UniqueName: \"kubernetes.io/projected/660bd85f-78bb-418d-aae8-6e888a8f7b31-kube-api-access-tbb8p\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.271616 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.271489 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.271616 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.271533 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660bd85f-78bb-418d-aae8-6e888a8f7b31-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.271616 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.271558 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.271616 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.271586 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.271833 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.271627 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.271833 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.271666 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.271833 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.271691 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.271992 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.271852 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.271992 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.271898 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.271992 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.271940 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.272145 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.272123 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.272183 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.272149 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.272513 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.272476 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660bd85f-78bb-418d-aae8-6e888a8f7b31-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.274540 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.274518 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.274668 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.274650 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.332600 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.332545 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660bd85f-78bb-418d-aae8-6e888a8f7b31-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.332795 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.332776 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbb8p\" (UniqueName: \"kubernetes.io/projected/660bd85f-78bb-418d-aae8-6e888a8f7b31-kube-api-access-tbb8p\") pod \"maas-default-gateway-openshift-default-6dcf757899-mc2nj\" (UID: \"660bd85f-78bb-418d-aae8-6e888a8f7b31\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.384160 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.384140 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:08.809151 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.809119 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj"] Nov 02 13:50:08.810325 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:50:08.810294 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660bd85f_78bb_418d_aae8_6e888a8f7b31.slice/crio-7d999eb5e9d41bada8d66a653737e7f1e502920a6536caa1c81add860051f3ee WatchSource:0}: Error finding container 7d999eb5e9d41bada8d66a653737e7f1e502920a6536caa1c81add860051f3ee: Status 404 returned error can't find the container with id 7d999eb5e9d41bada8d66a653737e7f1e502920a6536caa1c81add860051f3ee Nov 02 13:50:08.812390 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.812360 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:50:08.812452 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.812424 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:50:08.812490 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:08.812453 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:50:09.398059 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:09.398023 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" event={"ID":"660bd85f-78bb-418d-aae8-6e888a8f7b31","Type":"ContainerStarted","Data":"cfba959b81adc9ea9efd10089448fb3d7f6e7dc5c8ebe046a7eec52b09901900"} Nov 02 13:50:09.398444 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:09.398090 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" event={"ID":"660bd85f-78bb-418d-aae8-6e888a8f7b31","Type":"ContainerStarted","Data":"7d999eb5e9d41bada8d66a653737e7f1e502920a6536caa1c81add860051f3ee"} Nov 02 13:50:09.525329 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:09.525255 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" podStartSLOduration=1.5252402950000001 podStartE2EDuration="1.525240295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:09.525069716 +0000 UTC m=+6826.555305967" watchObservedRunningTime="2026-11-02 13:50:09.525240295 +0000 UTC m=+6826.555476545" Nov 02 13:50:10.385246 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:10.385210 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:10.386766 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:10.386740 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mc2nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 13:50:10.386905 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:10.386795 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" podUID="660bd85f-78bb-418d-aae8-6e888a8f7b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 13:50:11.384772 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:11.384735 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mc2nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 13:50:11.385154 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:11.384786 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" podUID="660bd85f-78bb-418d-aae8-6e888a8f7b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 13:50:12.385069 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:12.385034 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mc2nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 13:50:12.385472 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:12.385083 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" podUID="660bd85f-78bb-418d-aae8-6e888a8f7b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 13:50:13.385393 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:13.385356 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mc2nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 13:50:13.385810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:13.385410 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" podUID="660bd85f-78bb-418d-aae8-6e888a8f7b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 13:50:14.384640 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:14.384600 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mc2nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 13:50:14.384829 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:14.384666 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" podUID="660bd85f-78bb-418d-aae8-6e888a8f7b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 13:50:15.385183 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:15.385147 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mc2nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 13:50:15.385693 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:15.385206 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" podUID="660bd85f-78bb-418d-aae8-6e888a8f7b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 13:50:16.384764 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:16.384731 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mc2nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 13:50:16.384935 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:16.384791 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" podUID="660bd85f-78bb-418d-aae8-6e888a8f7b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 13:50:18.304022 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:18.303985 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:18.304516 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:18.304357 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:18.305219 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:18.305195 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" Nov 02 13:50:23.169397 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.169363 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8"] Nov 02 13:50:23.172589 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.172574 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.277249 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.277220 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8"] Nov 02 13:50:23.297926 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.297902 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw79r\" (UniqueName: \"kubernetes.io/projected/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-kube-api-access-dw79r\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.298080 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.297935 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.298080 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.297955 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.298080 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.297977 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.298195 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.298131 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.298195 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.298162 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.298269 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.298225 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.298269 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.298256 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.298347 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.298294 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399179 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399149 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399179 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399182 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399364 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399206 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399364 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399228 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399364 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399245 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399364 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399285 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw79r\" (UniqueName: \"kubernetes.io/projected/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-kube-api-access-dw79r\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399364 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399309 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399364 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399328 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399364 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399362 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399773 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399634 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399773 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399722 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399773 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399753 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399936 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399804 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.399998 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.399960 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.401676 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.401650 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.402034 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.402011 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.460721 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.460663 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.460825 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.460778 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw79r\" (UniqueName: \"kubernetes.io/projected/77eebe76-b5e2-4d1a-9da3-1272dc4b4fca-kube-api-access-dw79r\") pod \"maas-default-gateway-openshift-default-6dcf757899-5cmc8\" (UID: \"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.485608 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.485587 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:23.911373 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.911347 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8"] Nov 02 13:50:23.913743 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:50:23.913716 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77eebe76_b5e2_4d1a_9da3_1272dc4b4fca.slice/crio-56560e71f3d6b6603373aa6efc401841cd0181360b55415d55e7a870d407b8d4 WatchSource:0}: Error finding container 56560e71f3d6b6603373aa6efc401841cd0181360b55415d55e7a870d407b8d4: Status 404 returned error can't find the container with id 56560e71f3d6b6603373aa6efc401841cd0181360b55415d55e7a870d407b8d4 Nov 02 13:50:23.915842 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.915812 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:50:23.915925 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.915878 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:50:23.915925 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:23.915910 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:50:24.478002 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:24.477961 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" event={"ID":"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca","Type":"ContainerStarted","Data":"3b70707c7877c1f6878181625fe5b3b39df9652524faad9ce7e1c39373df6ab0"} Nov 02 13:50:24.478409 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:24.478027 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" event={"ID":"77eebe76-b5e2-4d1a-9da3-1272dc4b4fca","Type":"ContainerStarted","Data":"56560e71f3d6b6603373aa6efc401841cd0181360b55415d55e7a870d407b8d4"} Nov 02 13:50:24.486405 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:24.486384 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:24.487893 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:24.487873 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5cmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 13:50:24.487984 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:24.487915 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" podUID="77eebe76-b5e2-4d1a-9da3-1272dc4b4fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 13:50:24.686242 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:24.686176 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" podStartSLOduration=1.686163555 podStartE2EDuration="1.686163555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:24.685286964 +0000 UTC m=+6841.715523214" watchObservedRunningTime="2026-11-02 13:50:24.686163555 +0000 UTC m=+6841.716399876" Nov 02 13:50:25.486697 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:25.486666 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5cmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 13:50:25.487089 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:25.486725 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" podUID="77eebe76-b5e2-4d1a-9da3-1272dc4b4fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 13:50:26.486370 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:26.486333 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5cmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 13:50:26.486569 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:26.486396 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" podUID="77eebe76-b5e2-4d1a-9da3-1272dc4b4fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 13:50:27.486909 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:27.486872 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5cmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 13:50:27.487321 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:27.486922 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" podUID="77eebe76-b5e2-4d1a-9da3-1272dc4b4fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 13:50:28.486875 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:28.486839 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5cmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 13:50:28.487059 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:28.486893 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" podUID="77eebe76-b5e2-4d1a-9da3-1272dc4b4fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 13:50:29.393090 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:50:29.393054 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:50:29.486209 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:29.486178 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5cmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 13:50:29.486386 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:29.486231 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" podUID="77eebe76-b5e2-4d1a-9da3-1272dc4b4fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 13:50:30.486030 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:30.485995 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5cmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 13:50:30.486393 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:30.486043 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" podUID="77eebe76-b5e2-4d1a-9da3-1272dc4b4fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 13:50:31.486768 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:31.486731 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5cmc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Nov 02 13:50:31.487145 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:31.486791 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" podUID="77eebe76-b5e2-4d1a-9da3-1272dc4b4fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Nov 02 13:50:33.227585 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:33.227547 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:33.228063 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:33.227902 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:33.228745 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:33.228726 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" Nov 02 13:50:42.963076 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:42.963031 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:53.053836 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:53.053788 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:53.311542 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:50:53.311450 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:50:59.373447 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:50:59.373401 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:51:29.371904 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:51:29.371830 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:51:30.274516 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:30.274468 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:51:36.462526 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:36.462474 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:51:43.294938 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:43.294898 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:51:53.894998 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:53.894958 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r"] Nov 02 13:51:53.898439 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:53.898420 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.050770 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.050737 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.050932 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.050785 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.050932 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.050886 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.050932 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.050916 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.051060 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.050978 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.051060 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.050999 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.051060 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.051021 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcb45\" (UniqueName: \"kubernetes.io/projected/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-kube-api-access-tcb45\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.051177 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.051167 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.051213 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.051202 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.103276 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.103244 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r"] Nov 02 13:51:54.151723 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.151650 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.151723 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.151686 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.151723 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.151705 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.151969 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.151735 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.151969 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.151759 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.151969 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.151786 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcb45\" (UniqueName: \"kubernetes.io/projected/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-kube-api-access-tcb45\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.151969 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.151839 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.151969 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.151881 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.151969 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.151919 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.152281 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.152071 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.152281 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.152179 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.152281 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.152244 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.152441 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.152330 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.152597 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.152574 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.154228 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.154204 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.154527 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.154489 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.212131 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.212110 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.212338 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.212318 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcb45\" (UniqueName: \"kubernetes.io/projected/cd25f8fa-05f7-4c6f-bee7-70783a7ee634-kube-api-access-tcb45\") pod \"maas-default-gateway-openshift-default-6dcf757899-cn28r\" (UID: \"cd25f8fa-05f7-4c6f-bee7-70783a7ee634\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.510829 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.510798 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:54.934047 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.934023 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r"] Nov 02 13:51:54.935992 ip-10-0-141-170 kubenswrapper[4312]: W1102 13:51:54.935965 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd25f8fa_05f7_4c6f_bee7_70783a7ee634.slice/crio-50c08f13201395226dfd21c9a2c36036475ab6e3d91c1f3ab3ba9f04840680bd WatchSource:0}: Error finding container 50c08f13201395226dfd21c9a2c36036475ab6e3d91c1f3ab3ba9f04840680bd: Status 404 returned error can't find the container with id 50c08f13201395226dfd21c9a2c36036475ab6e3d91c1f3ab3ba9f04840680bd Nov 02 13:51:54.937970 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.937935 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:51:54.938040 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.937996 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:51:54.938040 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:54.938035 4312 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 13:51:55.888889 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:55.888851 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" event={"ID":"cd25f8fa-05f7-4c6f-bee7-70783a7ee634","Type":"ContainerStarted","Data":"1a536077ecd4f6b9eb1c54f887004dd870e289c96ec0903e56ee62c1aa55d165"} Nov 02 13:51:55.889076 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:55.888909 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" event={"ID":"cd25f8fa-05f7-4c6f-bee7-70783a7ee634","Type":"ContainerStarted","Data":"50c08f13201395226dfd21c9a2c36036475ab6e3d91c1f3ab3ba9f04840680bd"} Nov 02 13:51:56.078845 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:56.078797 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" podStartSLOduration=3.078784235 podStartE2EDuration="3.078784235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:51:56.060734106 +0000 UTC m=+6933.090970360" watchObservedRunningTime="2026-11-02 13:51:56.078784235 +0000 UTC m=+6933.109020484" Nov 02 13:51:56.511758 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:56.511729 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:51:56.513280 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:56.513252 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cn28r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 13:51:56.513408 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:56.513308 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" podUID="cd25f8fa-05f7-4c6f-bee7-70783a7ee634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 13:51:57.511154 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:57.511118 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cn28r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 13:51:57.511570 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:57.511170 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" podUID="cd25f8fa-05f7-4c6f-bee7-70783a7ee634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 13:51:58.511518 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:58.511471 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cn28r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 13:51:58.511911 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:58.511545 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" podUID="cd25f8fa-05f7-4c6f-bee7-70783a7ee634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 13:51:59.393302 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:51:59.393254 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:51:59.511421 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:59.511387 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cn28r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 13:51:59.511593 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:51:59.511443 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" podUID="cd25f8fa-05f7-4c6f-bee7-70783a7ee634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 13:52:00.511867 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:00.511832 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cn28r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 13:52:00.512325 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:00.511883 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" podUID="cd25f8fa-05f7-4c6f-bee7-70783a7ee634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 13:52:01.511621 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:01.511592 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cn28r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 13:52:01.511795 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:01.511641 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" podUID="cd25f8fa-05f7-4c6f-bee7-70783a7ee634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 13:52:02.511786 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:02.511754 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cn28r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 02 13:52:02.512173 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:02.511801 4312 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" podUID="cd25f8fa-05f7-4c6f-bee7-70783a7ee634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 02 13:52:04.065103 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:04.065058 4312 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:52:04.065797 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:04.065736 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:52:04.067000 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:04.066981 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" Nov 02 13:52:29.371948 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:52:29.371910 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:52:40.438075 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:40.437975 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:40.688001 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:40.687965 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:43.690779 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:52:43.690740 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:52:59.373448 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:52:59.373409 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:53:00.872266 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:00.872223 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:01.621440 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:01.621408 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:01.886328 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:01.886251 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:06.193612 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:06.193570 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:06.457792 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:06.457701 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:09.552964 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:09.552920 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:09.804321 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:09.804242 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:12.564727 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:12.564696 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:12.851226 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:12.851146 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:17.693363 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:17.693322 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:22.632247 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:22.632206 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:27.752528 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:27.752473 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:29.371874 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:53:29.371834 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:53:30.961125 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:30.961090 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:31.776848 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:31.776818 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:32.035427 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:32.035313 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:32.402627 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:32.402548 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:32.955207 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:32.955172 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:37.918018 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:37.917976 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:38.736129 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:38.736095 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:42.880714 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:42.880676 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:43.993933 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:43.993895 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:47.785538 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:47.785488 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:52.709351 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:52.709317 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:57.724467 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:53:57.724434 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:53:59.373490 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:53:59.373442 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:54:02.598407 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:02.598363 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:07.489111 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:07.489030 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:12.498912 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:12.498871 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:17.403619 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:17.403583 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:22.305351 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:22.305319 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:27.205796 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:27.205759 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:29.373103 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:54:29.373064 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:54:30.751640 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:30.751612 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:31.006358 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:31.006278 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:32.200767 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:32.200734 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:37.101300 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:37.101268 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:42.097809 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:42.097771 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:44.336583 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:44.336549 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:44.585904 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:44.585877 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:44.852599 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:44.852569 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:46.990913 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:46.990881 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:51.976902 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:51.976858 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:56.905243 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:54:56.905208 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:54:59.373650 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:54:59.373610 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:55:01.798026 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:01.797993 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:03.168746 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:03.168715 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:06.718526 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:06.718468 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:09.462119 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:09.462085 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:09.705041 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:09.705009 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:09.954343 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:09.954312 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:11.147475 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:11.147441 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:16.079103 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:16.079060 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:22.069810 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:22.069771 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:22.334738 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:22.334660 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:23.762378 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:23.762341 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:24.432583 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:24.432551 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:29.372282 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:55:29.372246 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:55:44.773792 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:55:44.773737 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:55:59.372275 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:55:59.372234 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:56:29.240167 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:56:29.240136 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:29.372192 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:56:29.372156 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:56:29.516536 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:56:29.516460 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:38.101676 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:56:38.101641 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:39.760776 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:56:39.760734 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:45.122167 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:56:45.122122 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:54.900715 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:56:54.900678 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:55.146829 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:56:55.146787 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:56:59.434758 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:56:59.433392 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:57:29.170433 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:57:29.170395 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:29.372795 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:57:29.372760 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:57:29.426331 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:57:29.426266 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:45.536410 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:57:45.536283 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:57:59.376391 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:57:59.376346 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:58:13.584835 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:58:13.584789 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:13.912573 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:58:13.912447 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:29.371642 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:58:29.371604 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:58:45.916539 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:58:45.916320 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:48.060332 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:58:48.060299 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:49.667519 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:58:49.667075 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:58:50.120141 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:58:50.119624 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cn28r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 13:58:50.120141 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:58:50.119697 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" podUID="cd25f8fa-05f7-4c6f-bee7-70783a7ee634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 13:58:59.374785 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:58:59.374744 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:59:29.370991 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:59:29.370954 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 13:59:46.228661 ip-10-0-141-170 kubenswrapper[4312]: I1102 13:59:46.228618 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 13:59:59.375130 ip-10-0-141-170 kubenswrapper[4312]: E1102 13:59:59.375087 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 14:00:02.362968 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:00:02.362894 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:02.625180 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:00:02.625099 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:29.372479 ip-10-0-141-170 kubenswrapper[4312]: E1102 14:00:29.372445 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 14:00:34.272521 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:00:34.271569 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-5cmc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:34.272521 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:00:34.271633 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-5cmc8" podUID="77eebe76-b5e2-4d1a-9da3-1272dc4b4fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:34.346624 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:00:34.346596 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-mc2nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:00:34.346783 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:00:34.346659 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-mc2nj" podUID="660bd85f-78bb-418d-aae8-6e888a8f7b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:00:46.599074 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:00:46.599035 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 14:00:59.373179 ip-10-0-141-170 kubenswrapper[4312]: E1102 14:00:59.373143 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 14:01:05.116521 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:01:05.115593 4312 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cn28r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 14:01:05.116521 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:01:05.115655 4312 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cn28r" podUID="cd25f8fa-05f7-4c6f-bee7-70783a7ee634" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 14:01:29.373065 ip-10-0-141-170 kubenswrapper[4312]: E1102 14:01:29.373027 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 14:01:43.725214 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:01:43.725172 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:43.987745 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:01:43.987661 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:46.935187 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:01:46.935147 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 14:01:59.373378 ip-10-0-141-170 kubenswrapper[4312]: E1102 14:01:59.373341 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 14:02:03.430539 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:03.430486 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vfxbz_fdf81983-c2ed-4444-9837-16ffca122340/manager/0.log" Nov 02 14:02:04.124765 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:04.124735 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7w4zh_5ced755a-4baf-4b4b-96df-f8a57839c593/server/0.log" Nov 02 14:02:05.931968 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:05.931938 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6459565456-qf5nj_be7afc4c-b5cd-4f04-ba9c-e688826371e8/maas-api/0.log" Nov 02 14:02:12.212636 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:12.212607 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wqxtt_39bf2d16-2a6d-4822-8d9c-b768de423729/manager/0.log" Nov 02 14:02:12.677117 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:12.677090 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-ck9gr_89fbd2ad-bb31-409d-90ba-b7d358be49a7/kuadrant-console-plugin/0.log" Nov 02 14:02:15.257606 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:15.257576 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85ccdd577f-llvhl_01362f25-65e7-4ec6-afcc-aa1410815d17/kube-auth-proxy/0.log" Nov 02 14:02:15.698000 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:15.697976 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-5cmc8_77eebe76-b5e2-4d1a-9da3-1272dc4b4fca/istio-proxy/0.log" Nov 02 14:02:15.926478 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:15.926442 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-cn28r_cd25f8fa-05f7-4c6f-bee7-70783a7ee634/istio-proxy/0.log" Nov 02 14:02:16.836710 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:16.836683 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-mc2nj_660bd85f-78bb-418d-aae8-6e888a8f7b31/istio-proxy/0.log" Nov 02 14:02:29.372309 ip-10-0-141-170 kubenswrapper[4312]: E1102 14:02:29.372268 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 14:02:33.525165 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:33.525129 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:33.783623 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:33.783542 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:41.717652 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:41.717608 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:42.225335 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:42.225302 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d5dpl_837a6c92-ea3e-490a-810e-05e2fc91a89a/global-pull-secret-syncer/0.log" Nov 02 14:02:45.393773 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:45.393740 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" Nov 02 14:02:46.413583 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:46.413555 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vfxbz_fdf81983-c2ed-4444-9837-16ffca122340/manager/0.log" Nov 02 14:02:46.774559 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:46.774475 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7w4zh_5ced755a-4baf-4b4b-96df-f8a57839c593/server/0.log" Nov 02 14:02:47.299388 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:47.299345 4312 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5pnd2_fcda3ab4-1f37-49db-8eb9-2b83fe89d484/konnectivity-agent/0.log" limit=1024 Nov 02 14:02:47.516389 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:47.516356 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-170.us-west-1.compute.internal_1ebeef05fd06ac44ab33873433450b74/haproxy/0.log" Nov 02 14:02:51.554373 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:51.554340 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vfxbz_fdf81983-c2ed-4444-9837-16ffca122340/manager/0.log" Nov 02 14:02:51.911709 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:51.911643 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7w4zh_5ced755a-4baf-4b4b-96df-f8a57839c593/server/0.log" Nov 02 14:02:52.855714 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:52.855676 4312 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm"] Nov 02 14:02:52.859299 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:52.859275 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:52.922362 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:52.922331 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm"] Nov 02 14:02:52.939667 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:52.939638 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-podres\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:52.939667 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:52.939666 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-lib-modules\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:52.939864 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:52.939688 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-sys\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:52.939864 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:52.939740 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-proc\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:52.939864 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:52.939832 4312 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brjfg\" (UniqueName: \"kubernetes.io/projected/205a6652-75ca-4642-abe1-a2b1bbd92f54-kube-api-access-brjfg\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.026968 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.026940 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z927c\"/\"kube-root-ca.crt\"" Nov 02 14:02:53.027108 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.026989 4312 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z927c\"/\"openshift-service-ca.crt\"" Nov 02 14:02:53.027108 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.027063 4312 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z927c\"/\"default-dockercfg-b99r6\"" Nov 02 14:02:53.040405 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.040383 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-sys\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.040486 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.040411 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-proc\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.040486 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.040437 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brjfg\" (UniqueName: \"kubernetes.io/projected/205a6652-75ca-4642-abe1-a2b1bbd92f54-kube-api-access-brjfg\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.040604 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.040530 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-proc\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.040604 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.040532 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-sys\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.040604 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.040554 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-podres\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.040604 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.040590 4312 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-lib-modules\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.040747 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.040610 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-podres\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.040747 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.040671 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/205a6652-75ca-4642-abe1-a2b1bbd92f54-lib-modules\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.101417 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.101393 4312 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brjfg\" (UniqueName: \"kubernetes.io/projected/205a6652-75ca-4642-abe1-a2b1bbd92f54-kube-api-access-brjfg\") pod \"perf-node-gather-daemonset-n9tzm\" (UID: \"205a6652-75ca-4642-abe1-a2b1bbd92f54\") " pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.171469 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.171445 4312 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:53.407918 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.407888 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wqxtt_39bf2d16-2a6d-4822-8d9c-b768de423729/manager/0.log" Nov 02 14:02:53.542009 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:53.541956 4312 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm"] Nov 02 14:02:53.544268 ip-10-0-141-170 kubenswrapper[4312]: W1102 14:02:53.544238 4312 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod205a6652_75ca_4642_abe1_a2b1bbd92f54.slice/crio-23129f52978afc9e9866d5eb059d930e343d34f3035900ef2f271cfa85d32383 WatchSource:0}: Error finding container 23129f52978afc9e9866d5eb059d930e343d34f3035900ef2f271cfa85d32383: Status 404 returned error can't find the container with id 23129f52978afc9e9866d5eb059d930e343d34f3035900ef2f271cfa85d32383 Nov 02 14:02:54.241912 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:54.241887 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-ck9gr_89fbd2ad-bb31-409d-90ba-b7d358be49a7/kuadrant-console-plugin/0.log" Nov 02 14:02:54.360332 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:54.360296 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" event={"ID":"205a6652-75ca-4642-abe1-a2b1bbd92f54","Type":"ContainerStarted","Data":"87e8136aa05fd97e9ed459c1e4d5d37d62852eeef5f53f48cc753c52acb5d4da"} Nov 02 14:02:54.360492 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:54.360343 4312 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" event={"ID":"205a6652-75ca-4642-abe1-a2b1bbd92f54","Type":"ContainerStarted","Data":"23129f52978afc9e9866d5eb059d930e343d34f3035900ef2f271cfa85d32383"} Nov 02 14:02:54.360492 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:54.360381 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:02:54.477262 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:54.477209 4312 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" podStartSLOduration=2.477195197 podStartE2EDuration="2.477195197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:54.475786229 +0000 UTC m=+7591.506022479" watchObservedRunningTime="2026-11-02 14:02:54.477195197 +0000 UTC m=+7591.507431447" Nov 02 14:02:56.952139 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:56.952091 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vfxbz_fdf81983-c2ed-4444-9837-16ffca122340/manager/0.log" Nov 02 14:02:57.660902 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:02:57.660875 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7w4zh_5ced755a-4baf-4b4b-96df-f8a57839c593/server/0.log" Nov 02 14:02:59.375197 ip-10-0-141-170 kubenswrapper[4312]: E1102 14:02:59.375161 4312 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p7qcg" containerName="extractor" Nov 02 14:03:00.374244 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:00.374214 4312 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z927c/perf-node-gather-daemonset-n9tzm" Nov 02 14:03:05.466533 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:05.466486 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-85ccdd577f-llvhl_01362f25-65e7-4ec6-afcc-aa1410815d17/kube-auth-proxy/0.log" Nov 02 14:03:05.905612 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:05.905487 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-5cmc8_77eebe76-b5e2-4d1a-9da3-1272dc4b4fca/istio-proxy/0.log" Nov 02 14:03:06.120324 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:06.120296 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-cn28r_cd25f8fa-05f7-4c6f-bee7-70783a7ee634/istio-proxy/0.log" Nov 02 14:03:06.279669 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:06.279645 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvcvg_1bb27c13-1527-4670-ae73-b64d4ef2ef0b/node-exporter/0.log" Nov 02 14:03:06.698447 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:06.698416 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvcvg_1bb27c13-1527-4670-ae73-b64d4ef2ef0b/kube-rbac-proxy/0.log" Nov 02 14:03:06.986448 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:06.986381 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-mc2nj_660bd85f-78bb-418d-aae8-6e888a8f7b31/istio-proxy/0.log" Nov 02 14:03:07.112567 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:07.112544 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tvcvg_1bb27c13-1527-4670-ae73-b64d4ef2ef0b/init-textfile/0.log" Nov 02 14:03:07.552433 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:07.552405 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bw6tl_bb36a3de-e4d1-4be1-b4c9-608e882fa7d9/kube-rbac-proxy-main/0.log" Nov 02 14:03:07.973849 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:07.973819 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bw6tl_bb36a3de-e4d1-4be1-b4c9-608e882fa7d9/kube-rbac-proxy-self/0.log" Nov 02 14:03:08.389800 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:08.389729 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-bw6tl_bb36a3de-e4d1-4be1-b4c9-608e882fa7d9/openshift-state-metrics/0.log" Nov 02 14:03:08.815844 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:08.815814 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb124f89-493d-435b-b51c-a78351f17aa1/prometheus/0.log" Nov 02 14:03:09.228977 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:09.228948 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb124f89-493d-435b-b51c-a78351f17aa1/config-reloader/0.log" Nov 02 14:03:09.643265 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:09.643197 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb124f89-493d-435b-b51c-a78351f17aa1/thanos-sidecar/0.log" Nov 02 14:03:10.058932 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:10.058909 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb124f89-493d-435b-b51c-a78351f17aa1/kube-rbac-proxy-web/0.log" Nov 02 14:03:10.474489 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:10.474462 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb124f89-493d-435b-b51c-a78351f17aa1/kube-rbac-proxy/0.log" Nov 02 14:03:10.913921 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:10.913899 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb124f89-493d-435b-b51c-a78351f17aa1/kube-rbac-proxy-thanos/0.log" Nov 02 14:03:11.328688 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:11.328603 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fb124f89-493d-435b-b51c-a78351f17aa1/init-config-reloader/0.log" Nov 02 14:03:11.756179 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:11.756140 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ttsrt_f52c28f9-63be-48b7-9643-fac197099856/prometheus-operator/0.log" Nov 02 14:03:12.164611 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:12.164586 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-ttsrt_f52c28f9-63be-48b7-9643-fac197099856/kube-rbac-proxy/0.log" Nov 02 14:03:12.581098 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:12.581026 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-f72f5_af313ed2-a9bc-4b05-a16c-5b0280657674/prometheus-operator-admission-webhook/0.log" Nov 02 14:03:12.920732 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:12.920707 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-wqxtt_39bf2d16-2a6d-4822-8d9c-b768de423729/manager/0.log" Nov 02 14:03:13.345066 ip-10-0-141-170 kubenswrapper[4312]: I1102 14:03:13.344991 4312 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-ck9gr_89fbd2ad-bb31-409d-90ba-b7d358be49a7/kuadrant-console-plugin/0.log"